Need two different versions of checkpointing; restart checkpoints will be sharded with the intent of reloading into a deepspeed trainer and need to include optimizer states. Best Val checkpoints are saved as single models (no sharding), do not require optimizer states. These checkpoints are intended as the 'inference pipeline' checkpoints.
Need two different versions of checkpointing; restart checkpoints will be sharded with the intent of reloading into a deepspeed trainer and need to include optimizer states. Best Val checkpoints are saved as single models (no sharding), do not require optimizer states. These checkpoints are intended as the 'inference pipeline' checkpoints.