Create Automated Training Pipeline

GeroVanMi / algorithmic-quartet-mlops

A showcase Machine Learning Operations (MLOps) Project.

0 stars 1 forks source link

Closed GeroVanMi closed 5 months ago

GeroVanMi commented 6 months ago

[x] Run GitHub Action when the tag run_training is pushed
[x] We are running out of space with the Pytorch image, so let's switch to a Nvidia / CUDA based one
[x] Still running out of space, so perhaps switch back to the normal docker build command?
[x] Implement Google Cloud Build pipeline
[x] Create secrets for W&B and Cloud Bucket keys
[x] Start Lightning AI studio with GPU and run training
[x] Track metrics to W&B project #29
[x] #12

GeroVanMi commented 5 months ago

Note: When running a docker container on Lightning, we need to pass the --gpus all flag, otherwise it won't be able to access the GPU!

docker run -it --gpus all IMAGE_NAME

GeroVanMi commented 5 months ago

This isn't quite as straightforward as I would like it to be mostly because we run out of space on the GitHub Actions Runner.

GeroVanMi commented 5 months ago

Completed.