We use a pre-trained resnet in our encoder. The pre-trained weights are getting downloaded on-demand when we use the model for the first time. Usually this is fine when working without docker. With docker the container will get shut down after each command invokation which means we have to download the model weights every time we run a new docker run .. train command.
This changset instead pre-fetches the pre-trained weights and bakes them into the docker images at
~/.cache/torch/checkpoints/resnet50-19c8e357.pth
Then docker run .. train never has to download the weights on-demand, we are no longer dependent on external services, and everything works beautifully.
The downside is the docker image size will increase since the weights are ~100 MB. But because the nvidia docker image is a few GBs anyway I don't see a problem here.
@jacquestardie merge please! Tested this on my rig!
We use a pre-trained resnet in our encoder. The pre-trained weights are getting downloaded on-demand when we use the model for the first time. Usually this is fine when working without docker. With docker the container will get shut down after each command invokation which means we have to download the model weights every time we run a new
docker run .. train
command.This changset instead pre-fetches the pre-trained weights and bakes them into the docker images at
Then
docker run .. train
never has to download the weights on-demand, we are no longer dependent on external services, and everything works beautifully.The downside is the docker image size will increase since the weights are ~100 MB. But because the nvidia docker image is a few GBs anyway I don't see a problem here.
@jacquestardie merge please! Tested this on my rig!