search

geopavlakos / hamer

HaMeR: Reconstructing Hands in 3D with Transformers
https://geopavlakos.github.io/hamer/
MIT License
352 stars 30 forks source link

How to train with multi GPUs？ #64

Closed YHaooo-4508 closed 3 weeks ago

YHaooo-4508 commented 3 weeks ago

I got the error when train with multi GPUs.

RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step. If this is intentional, you must enable the detection of unused parameters in DDP, either by setting the string value strategy='ddp_find_unused_parameters_true' or by setting the flag in the strategy with strategy=DDPStrategy(find_unused_parameters=True).

Using strategy: ddp_find_unused_parameters_true can solve this error, but the training speed is very slow.

Could you please tell me how to solve this problem or how to train with muti GPUs?

geopavlakos commented 3 weeks ago

When running on multiple GPUs, we also use the strategy ddp_find_unused_parameters_true during training.