Open coreylowman opened 2 years ago
It's also good to have a way to clone in memory the state of optimizer. It's useful if you want to pit 2 networks to see if the trained network performs better than the one before training (which is part of alpha go algorithm).
@kurnevsky thanks for bringing that up, cloning optimizers is actually really easy with some changes recently. Addressed in the above PR 👍
A common feature to use when building training frameworks is to save the optimizer state along with network state.
There's already a way to convert a
D::Vec<E>
into a rustVec<E>
, which should make it easy to save Gradients object. However a pain point is that gradients are currently keyed off of UniqueId, which may not be persistent across saving/loading depending on how model is initialized.Options:
HashMap<String, D::Vec<E>>
where String would be the full path to the tensor. This would make it easy to use with TensorCollection, and also require minimal changes to implement this serialization.