Lightning-AI / pytorch-lightning

Pretrain, finetune ANY AI model of ANY size on multiple GPUs, TPUs with zero code changes.
https://lightning.ai
Apache License 2.0
28.47k stars 3.39k forks source link

Adapt `find_unused_parameters` warning #12445

Closed carmocca closed 1 year ago

carmocca commented 2 years ago

🚀 Feature

Latter PyTorch versions introduced the following warning

https://github.com/pytorch/pytorch/blob/d0adb5ff264df8e0e057ce1178feb30198c601d2/torch/csrc/distributed/c10d/reducer.cpp#L1256-L1264

which appears when find_unused_parameters=True but it's not necessary.

Motivation

We set find_unused_parameters=True by default, so this warning will appear for a large percentage of users.

Pitch

Catch this warning and produce another one that explicitly mentions how to modify the strategy passed.


If you enjoy Lightning, check out our other projects! âš¡

cc @borda @akihironitta

carmocca commented 2 years ago

This is blocked by https://github.com/pytorch/pytorch/issues/75725