search

JulesBelveze / bert-squeeze

🛠️ Tools for Transformers compression using PyTorch Lightning ⚡
https://julesbelveze.github.io/bert-squeeze/
78 stars 10 forks source link

Error in `DataModule` when running on multiple GPUs with `ddp` #6

Closed JulesBelveze closed 1 year ago

JulesBelveze commented 2 years ago 
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
[2021-10-27 06:28:58,303][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 1
[2021-10-27 06:28:58,316][torch.distributed.distributed_c10d][INFO] - Added key: store_based_barrier_key:1 to store for rank: 0
[2021-10-27 06:28:58,318][torch.distributed.distributed_c10d][INFO] - Rank 0: Completed store-based barrier for 2 nodes.
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All DDP processes registered. Starting ddp with 2 processes
----------------------------------------------------------------------------------------------------

[2021-10-27 06:28:58,324][torch.distributed.distributed_c10d][INFO] - Rank 1: Completed store-based barrier for 2 nodes.
Error executing job with overrides: []
Traceback (most recent call last):
  File "/home/jules/bert-squeeze/bert-squeeze/main.py", line 58, in run
    trainer.fit(model, data)
  File "/home/jules/bert-squeeze/.venv/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 552, in fit
    self._run(model)
  File "/home/jules/bert-squeeze/.venv/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 864, in _run
    self._call_setup_hook(model)  # allow user to setup lightning_module in accelerator environment
  File "/home/jules/bert-squeeze/.venv/env/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1177, in _call_setup_hook
    self.datamodule.setup(stage=fn)
  File "/home/jules/bert-squeeze/.venv/env/lib/python3.6/site-packages/pytorch_lightning/core/datamodule.py", line 428, in wrapped_fn
    fn(*args, **kwargs)
  File "/home/jules/bert-squeeze/bert-squeeze/data/modules/transformer_module.py", line 62, in setup
    featurized_dataset = self.featurize()
  File "/home/jules/bert-squeeze/bert-squeeze/data/modules/transformer_module.py", line 45, in featurize
    tokenized_dataset = self.dataset.map(
AttributeError: 'NoneType' object has no attribute 'map'