search

lorenmt / reco

The implementation of "Bootstrapping Semantic Segmentation with Regional Contrast" [ICLR 2022].
https://shikun.io/projects/regional-contrast
Other
162 stars 25 forks source link

What should I do if I want to use reco loss on multiple GPUs. #27

Closed lyccol closed 2 years ago

lyccol commented 2 years ago

What should I do if I want to use reco loss on multiple GPUs.

lorenmt commented 2 years ago

I would suggest check out accelerate framework: https://github.com/huggingface/accelerate

lyccol commented 2 years ago

Thanks for your reply, I reimplemented a multi-GPU code using DDP.

Another problem is that I found that randomcrop will crop out the valid pixels in partial_p0.

This will cause reco loss = 0 when I train partial_p0 data. At this point I need to set find_unused_parameters = True to continue training, but this greatly affects the training efficiency. Do you have any suitable suggestions?

lorenmt commented 2 years ago

I thought ReCo should also use highly confident pseudo-labels, and considering we are learning in a batch-wise manner, so it shouldn't be that common to have invalid loss everywhere?