Sometimes things stop training e.g. application crashes, accidentally hitting ctrl-c on the wrong terminal, power outages, running out of memory etc. It would be handy to have the system save the memory buffer at fixed intervals and be able to load these back in to continue training if anything does go wrong.
beardyFace
commented
2 months ago
Sometimes things stop training e.g. application crashes, accidentally hitting ctrl-c on the wrong terminal, power outages, running out of memory etc. It would be handy to have the system save the memory buffer at fixed intervals and be able to load these back in to continue training if anything does go wrong.