Worker template default not set to tags

[delgadom]

During CI, we patch worker_template.yml to inject the default worker image to use so it's consistent with the notebook: https://github.com/RhodiumGroup/docker_images/blob/master/.travis.yml#L11

- "if [[ \"$TRAVIS_TAG\" == \"\" ]]; then
        sed -i.bak \
            's/image: rhodium\\/worker.*/image: rhodium\\/worker:'\"$TRAVIS_COMMIT\"'/' \
            notebook/worker-template.yml;
    else
        sed -i.bak \
            's/image: rhodium\\/worker:.*/image: rhodium\\/worker:'\"$TRAVIS_TAG\"'/' \
            notebook/worker-template.yml;
    fi"

I think this might not work well when we have multiple images as options because worker_template.yml only updates the version in /home/jovyan if it is more recent. This means (I think) that the defaults will only be updated once after each release, and then they'll get stuck there. Maybe we should always overwrite worker_template.yml? That's kind of annoying for people who want to change their defaults... hmm...

[delgadom]

We parse the worker_template.yml manually and then use KubeCluster.from_dict. Maybe we could store defaults in a different file (e.g. ~/.worker-defaults.yml) and then update them with anything the user puts in worker_template.yml?

[bolliger32]

Agreed with that plan @delgadom . I'm still a little confused as to why it wouldn't overwrite every time though, given current settings. If the file gets updated with a new worker commit hash, that would give it a new timestamp. So everytime prepare.sh runs, which should happen at the start of each session (i think?) it would overwrite the version in /home/jovyan if theres a new worker hash, right?

[bolliger32]

Oh wait i see your point. Right when you release the image, it will overwrite, but if a user then makes changes to it, then next time the notebook instance spins up, it will check the timestamps and see that the user-modified version is newer, so it won't copy over. So if someone is switching back and forth between images, that causes a problem. I like the worker-defaults plan, which doesn't get touched by users and which loads each time the notebook instance starts, with a separate worker-overrides file that is user-modified.

[delgadom]

Yep, and even if the user doesn't update the file, switching between image flavors will settle on the most recently published image (most recently modified worker-template.yml within the docker image) regardless of which notebook image is currently selected.