search

RhodiumGroup / rhg_compute_tools

Tools for using compute.rhg.com and compute.impactlab.org
MIT License
1 stars 4 forks source link

add tag argument to get_cluster #80

Closed delgadom closed 3 years ago

delgadom commented 4 years ago

Allows you to pass in a tag argument to get_cluster, which overrides just the tag of the default image in worker_template.yml. This is useful for pyTC images, where the base image is changed depending on the notebook you select, but the tag is dependent on the pyTC branch you're using.

delgadom commented 4 years ago

@bolliger32 I think this could help significantly with pyTC, as you could just pass in tag='branchname' as long as the dev-pyTC-base images were modified to set the default image in worker-template.yml to gcr.io/rhg-project-1/pytc-image[-basedev]

bolliger32 commented 4 years ago

yeah seems like a cool addition! I always forget what the gcr.io/rhg-project-1 prefix is and always end up doing ctrl-f in an earlier notebook to find it :) Couple things:

  1. I suppose we should update worker_template.yml in the dev-pyTC-base branch of docker_images so that at least the latest image gets the appropriate default image gcr.io/rhg-project-1/pytc-image-devbase:latest. So that would just be replacing rhodium/worker:WILL_BE_REPLACED_BY_TRAVIS with gcr.io/rhg-project-1/pytc-image-devbase:latest and then dropping the sed stuff in the travis yml.

  2. I think even with https://github.com/RhodiumGroup/docker_images/pull/130 we still have the issue where if someone loads any of the latest images and then goes back to a stable image, they will be stuck with the default worker from the latest image. I'm not sure if there's an easy way around that until the next round of updating stable images.

delgadom commented 4 years ago

For (1) I was actually thinking we would modify the sed section with gcr.io/rhg-project-1/pytc-image-devbase:$TRAVIS_COMMIT or tag or whatever on the pyTC dev base branch.

For (2) yep. Hoping to update both stable images soon. I've asked everyone on our team to give rhodium:latest a whirl with as many of their workflows as possible.

bolliger32 commented 4 years ago

Sounds good. The benefit of 1 being that if we if we tag the commit we can create a tagged image? I'll make that change now.

delgadom commented 4 years ago

oops. thanks for the catch! yeah made this a WIP because I haven't tested this live yet... thanks for saving me a round of debugging! Also yeah you're totally right for (1) it shouldn't matter, but yeah tagged commits would be dope as the default for e.g. pyTC stable down the line.

bolliger32 commented 3 years ago

Closed by #80