Pytorch emits an odd message which screws up the process communication

[johann-petrak]

Pytorch emits the message:

Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.

Since we use stdout/stderr pipes for process communication, this messes up things and makes the application crash.

[johann-petrak]

This may not actually be pytorch but either some GPU-related library used by pytorch or the Elmo model code we use

[johann-petrak]

Searching all of anaconda this only is found in one file: lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py

Why would we load or use that file though??

[johann-petrak]

OK, allennlp contains both ELMO and BERT so this could happen when allennlp is loaded.

[johann-petrak]

Allennlp loads the pretrained bert modeling when loading the Elmo module. This causes the message. The modeling.py I had was from a version of pytorch pretrained which used "print" to emit the message to standard output. This has been changed in version 0.6.2 to use the logger.

So in order to fix this:

• we need to manually install latest version of pretrained bert but at least version 0.6.2 after installing allennlp: git@github.com:huggingface/pytorch-pretrained-BERT.git
• NOTE: allennlp depends on pretrained bert version 0.6.0 or later
• make sure the logger for modeling.py is set to a level that ignores the message: getLogger("pytorch_pretrained_bert.modeling") ... setLevel(logging.CRITICAL)

[johann-petrak]

The TextClassCnnSingleElmo module should also check the version of pytorch-pretrained and maybe just refuse to work unless it finds at least 0.6.2

[johann-petrak]

For now, requiring version 0.6.2 or later of pytorch_pretrained_bert is an acceptable fix.