open-mmlab / mmpose

OpenMMLab Pose Estimation Toolbox and Benchmark.
https://mmpose.readthedocs.io/en/latest/
Apache License 2.0
5.84k stars 1.25k forks source link

Loading checkpoint is too slow #1283

Open bzfather opened 2 years ago

bzfather commented 2 years ago

I firstly trained the model on servers and now I try to load it on desktop (Windows with 3090 CUDA11.4 pytorch1.10) I ran the demo and it was stuck at “Loading checkpoints.....” shown in picture. It took 20 minutes to load. What happened? 20220401_154439

ly015 commented 2 years ago

Thanks for your feedback. Does it always happen, or randomly happen? I did notice that in Windows sometimes the program gets stuck and needs to click Enter to continue, and it happens not only to mmpose but also other programs like git. Not sure if it is related to your issue.

bzfather commented 2 years ago

Thanks for your feedback. Does it always happen, or randomly happen? I did notice that in Windows sometimes the program gets stuck and needs to click Enter to continue, and it happens not only to mmpose but also other programs like git. Not sure if it is related to your issue.

It always happened. It was stuck in this step when loading other checkpoints file as well. No reaction when clicking Enter. And when I checked task manager, I found that the memory of GPU increasing very slowly and only 2 cores of CPU working. After a long time of loading, it finished.

ly015 commented 2 years ago

It could be an MMCV issue. Could you please provide the environmental information by python -c "from mmcv.utils import collect_env; print(collect_env())"?