lukas / ml-class

Machine learning lessons and teaching projects designed for engineers
https://www.youtube.com/channel/UCBp3w4DCEC64FZr4k9ROxig
GNU General Public License v2.0
2.34k stars 1.17k forks source link

resolve OOM errors in transfer-learning or warn that it needs a GPU #67

Open charlesfrye opened 4 years ago

charlesfrye commented 4 years ago

On my reasonably-equipped home cpu-machine and on the (cpu) hub, the transfer-learning example causes OOM errors -- sometimes even before getting to the model.fit call.

They're pretty scary-looking, if you're not expecting them, and depending on the exact system parameters, they can sometimes be triggered inside of the wandb.init call, making it look like our fault.

Potential solutions:

charlesfrye commented 4 years ago

It also doesn't help that there are two copies of the data: one normalized and one not normalized.

Moving the normalization into the data-loading or into the model (as is standard Keras practice these days, to prevent GPUs from getting starved) would help.