search

real-stanford / xskill

[CoRL 2023] XSkill: cross embodiment skill discovery
https://xskill.cs.columbia.edu/
MIT License
48 stars 9 forks source link

Multi-GPU training #2

Open feriorior opened 9 months ago

feriorior commented 9 months ago

Thanks to the author for this excellent benchmark! When I try to change the GPU devices setting, I get the following error:

RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step. If this is intentional, you must enable the detection of unused parameters in DDP, either by setting the string valuestrategy='ddp_find_unused_parameters_true'or by setting the flag in the strategy withstrategy=DDPStrategy(find_unused_parameters=True).

It might be some parameter just used for forward but not backward, which might result in this error.

mengdaxu commented 9 months ago

Thank you for your feedback. We appreciate your interest in our benchmark. As of now, we haven't experimented with multi-GPU training. We will investigate this issue.