Unable to use bagua.torch_api.contrib.CachedDataset with PyTorchLightning

[chenzhekl]

🐛 Bug

We are unable to use bagua.torch_api.contrib.CachedDataset with PyTorchLightning. The creation of CachedDataset requires that the process group has been initialized.

Traceback (most recent call last):                                                        
  File "/home/zchen/workspace/project/src/train.py", line 181, in <module>  
    main()                                                                                
  File "/home/zchen/workspace/project/src/train.py", line 140, in main      
    dataset_train = CachedDataset(                                                        
  File "/opt/conda/lib/python3.8/site-packages/bagua/torch_api/contrib/cached_dataset.py",
 line 48, in __init__                                                                     
    self.cache_loader = CacheLoader(                                                      
  File "/opt/conda/lib/python3.8/site-packages/bagua/torch_api/contrib/cache_loader.py", l
ine 69, in __init__                                                                       
    self.store = RedisStore(**kwargs)                                                     
  File "/opt/conda/lib/python3.8/site-packages/bagua/torch_api/contrib/utils/redis_store.p
y", line 74, in __init__                                                                  
    hosts = bootstrap_redis_server(capacity_per_node)                                     
  File "/opt/conda/lib/python3.8/site-packages/bagua/torch_api/contrib/utils/redis_store.p
y", line 122, in bootstrap_redis_server                                                   
    default_store = c10d._get_default_store()                                             
  File "/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", lin
e 441, in _get_default_store                                                              
    raise RuntimeError(                                                                   
RuntimeError: Default process group has not been initialized, please make sure to call ini
t_process_group.

To Reproduce

Expected behavior

Sucess without errors.

Environment

• CUDA:
  • GPU:
    • NVIDIA A100 80GB PCIe
    • NVIDIA RTX A6000
    • NVIDIA RTX A6000
  • available: True
  • version: 11.3
• Packages:
  • numpy: 1.21.2
  • pyTorch_debug: False
  • pyTorch_version: 1.11.0
  • pytorch-lightning: 1.6.1
  • tqdm: 4.64.0
• System:
  • OS: Linux
  • architecture:
    • 64bit
    • ELF
  • processor: x86_64
  • python: 3.8.12
  • version: #40~20.04.1-Ubuntu SMP Mon Mar 7 09:18:32 UTC 2022

Additional context

cc @awaelchli @wangraying @akihironitta

[wangraying]

Yes, features in bagua contrib have not been supported in Pytorch Lightning yet.

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!