Distributed training hangs indefinitely at FP16_Optimizer#step

[justinmanley]

When I run distributed training with more than one GPU, training gets stuck at the very beginning and hangs indefinitely. It is stuck in FP16_Optimizer#set (specifically at this line, where data is implicitly moved from the GPU to the CPU).

The command line hangs at this indefinitely and makes no progress no matter how I wait:

training: 0%| | 0/1000000 [00:00<?, ?it/s]

I see this issue regardless of which CUDA version I use (I've tried it with 10.0 and 10.1), and regardless of whether I install apex myself or use the docker image icaruszyz/large-scale-training:dialogpt.

I do not experience this issue when I run demo.py rather than using python -m torch.distributed.launch to run training (i.e. I see this issue only when I try to train on multiple GPUs, not on a single GPU). I have not tried training with full 32-bit precision because I want to limit the number of GPUs I have to use.

The fact that this issue only occurs when training with multiple GPUs, and that it occurs on a line which transfers data from the GPU to the CPU, suggests to me that there may be a race condition related to collecting data from multiple GPUs.

Training configuration:

INFO - __main__ -   Input Argument Information
INFO - __main__ -   model_name_or_path            ./configs/762M
INFO - __main__ -   seed                          42
INFO - __main__ -   max_seq_length                128
INFO - __main__ -   skip_eval                     False
INFO - __main__ -   init_checkpoint               ./models/large/large_fs.pkl
INFO - __main__ -   train_input_file              ./data/train.128len.db
INFO - __main__ -   eval_input_file               ./data/dummy_data.tsv
INFO - __main__ -   continue_from                 0
INFO - __main__ -   train_batch_size              8
INFO - __main__ -   gradient_accumulation_steps   2
INFO - __main__ -   eval_batch_size               16
INFO - __main__ -   learning_rate                 0.0001
INFO - __main__ -   num_optim_steps               1000000
INFO - __main__ -   valid_step                    10000
INFO - __main__ -   warmup_proportion             0.1
INFO - __main__ -   warmup_steps                  16000
INFO - __main__ -   normalize_data                True
INFO - __main__ -   fp16                          True
INFO - __main__ -   lr_schedule                   noam
INFO - __main__ -   loss_scale                    0
INFO - __main__ -   no_token_id                   True
INFO - __main__ -   output_dir                    models/output_model
INFO - __main__ -   log_dir                       None
INFO - __main__ -   pbar                          True
INFO - __main__ -   local_rank                    0
INFO - __main__ -   config                        None
INFO - __main__ -   device                        cuda:0
INFO - __main__ -   n_gpu                         1

This bug is preventing me from fine-tuning the large model, which requires multiple GPUs.

Has anyone else experienced this or found a workaround?

[intersun]

Our model was trained on multiple GPU without issues, it might be a display issue related to progress bar, or different version of apex...

So can you try with a very small num_optim_steps (say 100) and see if it actually train the model or not?