DistributedSampler can't shuffle the dataset

[elk-cloner]

🐛 Bug

Information

I'm trying to fine-tune BERT model using run_language_modeling.py.

Language I am using the model on is Persian:

The problem arises when using:

• [x] the official example scripts: (give details below)
• [ ] my own modified scripts: (give details below)

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name)
• [x] my own task or dataset: (give details below)

But according to this issue there is a bug in torch.utils.data.distributed.DistributedSampler so that during different epochs shuffling operation doesn't work properly(it's not working). To solve this problem: according to pytorch official example here, we should add train_sampler.set_epoch(epoch) before each new epoch at this line

To reproduce

Steps to reproduce the behavior:

1. compare batches between different epoch like mentioned issue

Expected behavior

Environment info

• transformers version: transformers==2.8.0
• Platform: Ubuntu 18.04
• Python version: 3.7
• PyTorch version (GPU?): torch==1.4.0 (Yes)
• Tensorflow version (GPU?): tensorflow-gpu==2.1.0 (Yes)
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: distributed

[julien-c]

I think you are right

[vlevit]

Isn't there the same issue in other places? E.g. in trainer.py: https://github.com/huggingface/transformers/blob/97a375484c618496691982f62518130f294bb9a8/src/transformers/trainer.py#L305-L307

[julien-c]

I forgot to re-add this in Trainer when merging #3800

It's on my todo-list, but feel free to open a PR if you can do it faster than I can

[vlevit]

Great. Personally I've not yet upgraded to the newer version with trainer.py, so I'll leave it for you, thanks.