model-trainer-huggingface: resume from checkpoint

[samos123]

Testing using following resource:

apiVersion: substratus.ai/v1
kind: Model
metadata:
  name: falcon-7b-instruct-k8s
spec:
  image:
    name: substratusai/model-trainer-huggingface:pr-17
  baseModel:
    name: falcon-7b-instruct
  trainingDataset:
    name: k8s-instructions
  params:
    num_train_epochs: 1
    save_steps: 5
  resources:
    gpu:
      count: 4
      type: nvidia-l4

test case:

• Create model
• Wait for trainer to start and after step 40, kill the pod
• Check if pod gets restarted and see if starts from 40 isntead of 0

Note the huggingface default for save_steps is 500 which means it only stores checkpoints once every 500 steps: https://huggingface.co/docs/transformers/v4.31.0/en/main_classes/trainer#transformers.TrainingArguments.save_steps