Sub-workers exits without messages

[GongZhengLi]

🐛 Bug

I use the script as follow:

CUDA_VISIBLE_DEVICES="0, 1, 2, 3" metaseq-train --task streaming_language_modeling \ data/pile-test/ \ --num-workers 4 \ --reset-dataloader \ --vocab-filename ./vocab/gpt2-vocab.json \ --merges-filename ./vocab/gpt2-merges.txt \ --model-parallel-size 1 \ --ddp-backend fully_sharded \ --task-ddp-backend fully_sharded \ --criterion cross_entropy \ --batch-size 8 \ --save-dir /checkpoints/lm_transformer_pile-00 \ --arch transformer_lm_gpt2_tiny --share-decoder-input-output-embed \ --dropout 0.1 \ --optimizer adam --weight-decay 0.01 --clip-norm 0.0 \ --lr 0.0005 --lr-scheduler inverse_sqrt --warmup-updates 4000 --warmup-init-lr 1e-07 \ --tokens-per-sample 1024 --sample-break-mode none --fp16 \ --use-sharded-state \ --decoder-learned-pos \ --log-format json \ --log-interval 1

The rank 1, 2, 3 was exit before the loop of train_step. I print the every detailed log and find that the iter() inside more_itertools.peekable() kill all the non-master processes. What's the matter with this ?

[mahnerak]

I tried to go deeper and recover the errors causing exit. I ended up here: https://github.com/pytorch/pytorch/blob/db8abde9b6c4735d18d4681a1f70a55ff0b09f5b/torch/multiprocessing/spawn.py#L72-L76 Got an error:

Traceback (most recent call last):
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 69, in _wrap
    fn(i, *args)
  File "/home/USER/pip_editable_packages/metaseq/metaseq/distributed/utils.py", line 227, in distributed_main
    retval = main(cfg, **kwargs)
  File "/home/USER/pip_editable_packages/metaseq/metaseq/cli/train.py", line 181, in main
    valid_losses, should_stop = train(cfg, trainer, task, epoch_itr)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/USER/pip_editable_packages/metaseq/metaseq/cli/train.py", line 214, in train
    itr = epoch_itr.next_epoch_itr(
  File "/home/USER/pip_editable_packages/metaseq/metaseq/data/iterators.py", line 268, in next_epoch_itr
    self._itr = self._get_iterator_for_epoch(self.epoch)
  File "/home/USER/pip_editable_packages/metaseq/metaseq/data/iterators.py", line 395, in _get_iterator_for_epoch
    itr = StreamingCountingIterator(
  File "/home/USER/pip_editable_packages/metaseq/metaseq/data/iterators.py", line 125, in __init__
    self._peekable_itr = more_itertools.peekable(iterable)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/site-packages/more_itertools/more.py", line 311, in __init__
    self._it = iter(iterable)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 368, in __iter__
    return self._get_iterator()
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 314, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 927, in __init__
    w.start()
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/USER/miniconda3/envs/MY_METASEQ_ENV/lib/python3.9/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <enum 'Choices'>: attribute lookup Choices on metaseq.dataclass.constants failed

Which made workers sys.exit(1) immediately, without even letting them to send the error messages.

This is the command I launch the training.

metaseq-train 
--task streaming_language_modeling /home/USER/PROJECT/WORKDIR/pile/ 
--vocab-filename /home/USER/PROJECT/WORKDIR/vocab.json 
--merges-filename /home/USER/PROJECT/WORKDIR/merges.txt 
--criterion cross_entropy 
--batch-size 8 
--save-dir /home/USER/PROJECT/WORKDIR/ckpts/a4 
--arch transformer_lm 
--share-decoder-input-output-embed 
--dropout 0.1 
--optimizer adam 
--weight-decay 0.01 
--clip-norm 0.0 
--lr 0.0005 
--lr-scheduler inverse_sqrt 
--warmup-updates 4000 
--warmup-init-lr 1e-07 
--tokens-per-sample 1024 
--sample-break-mode none 
--decoder-learned-pos 
--log-format json 
--log-interval 1 
--aim-repo /home/USER/PROJECT/WORKDIR/. 
--save-interval-updates 30000 
--fp16

Two CUDA GPUs are available. Tested on both physical machine, VM, as well as Slurm.

Single-GPU version (just setting CUDA_VISIBLE_DEVICES env variable) works well.

[mahnerak]

The rank 1, 2, 3 was exit before the loop of train_step. I print the every detailed log and find that the iter() inside more_itertools.peekable() kill all the non-master processes.

Confirming that this error in my case too comes from more_itertools.peekable() so it's very likely we're experiencing with the same bug.

[GongZhengLi]

@mahnerak I solved this by add num_workers=0. It seems like a bug from pytorch !

[mahnerak]

Thanks @GongZhengLi

I don't think num_workers=0 will be okay in my setup. The data is too big. The training will be bottlenecked by data processing and the GPUs will be very underutilized :/

[GongZhengLi]

@mahnerak ， did you solve it ?

[mahnerak]

Not yet. Still waiting. I might come back to this issue in couple of days, but not sure if anything is changed related to this issue.

[jihwankwak]

@mahnerak @GongZhengLi DId any one solve this issue? I am facing the same issue while using 1 node & 2 gpus