search

project-lighter / lighter

Config-based framework for organized and reproducible deep learning. MONAI Bundle + PyTorch Lightning.
https://project-lighter.github.io/lighter
MIT License
21 stars 2 forks source link

Config instantiation is not DDP-aware #83

Open surajpaib opened 11 months ago

surajpaib commented 11 months ago

Because config is instantiated only on the main process and then DDP is initialized, objects that are initialized in the config are not DDP aware.

Maybe there is a way to fix this by instantiating trainer first and separate and then the system

ibro45 commented 7 months ago

I think this isn't an issue anymore, right? Closing.

surajpaib commented 4 months ago

@ibro45 This is still an issue.

Now I needed this for https://github.com/Lightning-AI/pytorch-lightning/discussions/11763

But since at instantiation, objects are not DDP aware, this problem persists.

ibro45 commented 4 months ago

Any idea on how to approach it?

ibro45 commented 2 months ago

Wait how is this a lighter problem? I think it has to do with improper usage/behavior of CacheDataset only

ibro45 commented 2 months ago

Would swapping the order of trainer and system help? https://github.com/project-lighter/lighter/blob/5d607684136d95434f2fee5aa5c38f1e32608335/lighter/utils/runner.py#L92