Open kuenishi opened 3 years ago
Just to get a heads up here, The issue is that the parameter buffers could be changed while we are writing them to storage?
Yes, exactly. We haven't have any precision issues so far, because storage is not so slow compared to training speed; 5-10 (iters/sec). But in case where training iteration is much faster than storage, it will be more problematic.
Looking around the snapshot code, I just found that they were almost ported from Chainer. That said, discussion in https://github.com/chainer/chainer/issues/6763 also applies to PPE. Just for heads up...