Getting hundreds of Warning messages

[XuhuiZhou]

Hi, when I run the following commands, allennlp train --file-friendly-logging --include-package kb.include_all \ training_config/downstream/wic.jsonnet -s OUTPUT_DIRECTORY

I get hundreds of Warning messages like:

/opt/conda/conda-bld/pytorch_1565287148058/work/aten/src/ATen/native/cuda/LegacyDefinitions.cpp:14: UserWarning: maskedfill received a mask with dtype torch.uint8, this behavior is now deprecated,please use a mask with dtype torch.bool instead.

Which is annoying, any insights on how to avoid this problem?

[XuhuiZhou]

Plus I also find this warning message:

2020-08-25 16:55:48,611 - WARNING - root - NaN or Inf found in input tensor.

Is this an expected behavior?

[matt-peters]

You can ignore the first warning. Fixing it requires finding the line of code that is calling masked_fill_ and changing the type of the mask to bool. The second warning isn't normal and suggests either training went off the rails somewhere (perhaps due to hyperparameter settings, e.g. too large of a learning rate, or too small of an epsilon in the various places it's used to avoid division by 0), or a bug somewhere.

[XuhuiZhou]

Thanks for the answer. As you pointed out, I also think "The second warning isn't normal and suggests either training went off the rails somewhere". However, it is really hard for me to locate the problem and the training results seems to be aligned with what's reported in the paper:

In this case, does that mean that I should probably just ignore the warning here?

[matt-peters]

Since it seems to be working, then you can probably safely ignore the warning.