I ran into some strange problems and want to share my solution here. When I ran the training script it did not use the GPUs. Another problem was that distributed training froze during startup. I could solve the problems by adding version numbers for torch and pytorch-ignite to requirements.txt. So if you run into similar problems try using explicit version numbers when setting up your python environment.
Here is the new content of my requirements.txt:
torch==1.8.1
pytorch-ignite==0.3.0
transformers==2.5.1
tensorboardX==1.8
tensorflow # for tensorboardX
spacy
I ran into some strange problems and want to share my solution here. When I ran the training script it did not use the GPUs. Another problem was that distributed training froze during startup. I could solve the problems by adding version numbers for torch and pytorch-ignite to requirements.txt. So if you run into similar problems try using explicit version numbers when setting up your python environment.
Here is the new content of my requirements.txt: