Add support for resuming from checkpoint

[jacob-morrison]

We should add support for resuming training partway through, e.g. if a job is preempted. I think most of the functionality is there, but we'll need to 1) save more often (every epoch will probably be wasteful depending on how often we get preempted) and 2) handle --resume_from_checkpoint well.

We'll need to properly handle resume_from_checkpoint at the start of training and when resuming partway through, which will most likely involve looking for a checkpoint manually first before letting the automagic huggingface code resume training. We'll also need to not overwrite the output folder.

[natolambert]

Links to make this:

• Accelerate docs -- need to use save_state https://huggingface.co/docs/accelerate/en/usage_guides/checkpoint
• Discussions
  • https://discuss.huggingface.co/t/saving-optimizer/8884/3
  • https://github.com/huggingface/transformers/issues/7198
  • https://github.com/huggingface/accelerate/issues/95
• Transformers source for resuming with Trainer https://github.com/huggingface/transformers/blob/080e14b24c8923e4bc18fcd54010fc7396c67bc0/src/transformers/trainer.py#L1903