Right now sheeprl save the state_dict of every model on checkpoint: this can be problematic when for example the model definition changes from when the checkpoint has been saved to when the checkpoint is loaded. As specified in the Fabric doc, it is better to save the entire models in the checkpoint, and resume them entirely when needed
belerico
commented
11 months ago
Right now sheeprl save the
state_dictof every model on checkpoint: this can be problematic when for example the model definition changes from when the checkpoint has been saved to when the checkpoint is loaded. As specified in the Fabric doc, it is better to save the entire models in the checkpoint, and resume them entirely when needed