Whether the training is going on normally

isl-org / lang-seg

Language-Driven Semantic Segmentation

MIT License

691 stars 84 forks source link

Whether the training is going on normally #28

Closed qliu520 closed 1 year ago

qliu520 commented 1 year ago

When I train the code, the multi-GPU allocation stops at the following location： " Resuming checkpoint None, exp_version=None initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/5 Resuming checkpoint None, exp_version=None initializing ddp: GLOBAL_RANK: 4, MEMBER: 5/5

" Is this training? Or something wrong ？

Boyiliee commented 1 year ago

Hi @qliu520 ,

Thanks for your interest in LSeg!

It seems that you are not loading all the GPUs, you might need to check you machine and system.

Hope this helps.

Best, Boyi