search

fdahle / antarctic_segmentation

4 stars 0 forks source link

Mixed precision training to reduce GPU memory use? #1

Open weiji14 opened 2 years ago

weiji14 commented 2 years ago

Hi there,

Just listened to your EGU talk just now, very cool project! Amazing that you can get such nice segmentation resuts on monochrome images!

You mentioned that you had some issues relates to fitting the full dataset in GPU memory. Have you considered trying something lke 16-bit training instead of 32-bit training? It would be a one-line change in Pytorch Lightning (https://pytorch-lightning.readthedocs.io/en/1.6.3/advanced/precision.html#fp16-mixed-precision), but for your case it might require some extra code changes. Happy to chat more if you're interested.

fdahle commented 2 years ago

Hi weiji14,

glad you liked my project! Thanks for the hint with using 16 Bit of 32 Bit. Didn't thought of this yet, but you're right, it could help to fit more data in the GPU. I will check it out, if I get it to work and how it affects my result. I will let you know if I have results/or stuff is not working :)