Things to do

[ivyleavedtoadflax]

• [x] Take arguments
• [x] Take external data
• [x] Integrate dvc
• [ ] MLFlow
• [ ] Save model and results objects
• [ ] Reduce Dockerfile size
• [ ] Lockdown permissions
• [ ] change bucket output
• [ ] Try running sagemaker through cml and github worfklow
• [ ] Spot instances (...and handle interrupted training)

[ivyleavedtoadflax]

Note to be able to restart after an interrupted spot instance training, you must check to see whether a checkpoint exists or not. We must also send the final data to the model, we can't rely on a final processing step (e.g. tokenisation, etc.) to be done in the container, it must receive the final data.

[ivyleavedtoadflax]

Need to pass environment variables to sagemaker containers to define MLFlow server