Open surajpaib opened 11 months ago
I think this isn't an issue anymore, right? Closing.
@ibro45 This is still an issue.
Now I needed this for https://github.com/Lightning-AI/pytorch-lightning/discussions/11763
But since at instantiation, objects are not DDP aware, this problem persists.
Any idea on how to approach it?
Wait how is this a lighter problem? I think it has to do with improper usage/behavior of CacheDataset only
Would swapping the order of trainer and system help? https://github.com/project-lighter/lighter/blob/5d607684136d95434f2fee5aa5c38f1e32608335/lighter/utils/runner.py#L92
Because config is instantiated only on the main process and then DDP is initialized, objects that are initialized in the config are not DDP aware.
Maybe there is a way to fix this by instantiating
trainer
first and separate and then thesystem