search

oceanhackweek / Hub-Management

General infrastructure management
0 stars 0 forks source link

Update the current base images to latest versions? #2

Open emiliom opened 1 year ago

emiliom commented 1 year ago

I'm not sure if this is needed as a base preparation for other updates to the OHW hub, but I'm putting it out here just in case.

abkfenris commented 1 year ago

The main thing that needs to be bumped would be pangeo-notebook as that controls the jupyter and other versions that should be kept in sync with the underlying hub.

https://github.com/oceanhackweek/jupyter-image/blob/99b6ab599e06104dae306f1dd73eaa89545db81b/py-base/environment.yml#L7

The base miniconda image probably could be bumped too.

emiliom commented 1 year ago

So, we modify py-base/environment.yml directly. I can submit a PR. Does that trigger a rebuild of the conda lock file?

The miniconda image is in https://github.com/oceanhackweek/jupyter-image/blob/99b6ab599e06104dae306f1dd73eaa89545db81b/py-base/Dockerfile#L2. I'd rather leave it to you to decide what's a good version to update to.

Once environmenty.yml and Dockerfile are updated, I see in .github/workflows/build-image.yml that the updated docker image is pushed to the GitHub Container Registry. Does that trigger a 2i2c hub image rebuild, or do we have to do a manual step somewhere to do that?

abkfenris commented 1 year ago

No, creating the conda-lock file has to be done manually: https://github.com/oceanhackweek/jupyter-image/issues/42

Preferably I'd rip miniconda out and use micromamba to make things more lightweight, but nb_conda_kernels doesn't understand that to allow our users to set up their own environments. gator may be able to take care of that, but that's a much deeper rabbit hole to go down. Just made myself and issue about that to dig into in the future: https://github.com/oceanhackweek/jupyter-image/issues/62

2i2c has an action that detects new images that runs periodically. It bumps the images in https://github.com/2i2c-org/infrastructure/blob/1c88f426c2f58c2d13060a9737fce5cbe5bc8f5a/config/clusters/2i2c/ohw.values.yaml#L38 They had us on a fast cycle for the last event, but they may have turned things down to weekly for us.

emiliom commented 1 year ago

No, creating the conda-lock file has to be done manually: https://github.com/oceanhackweek/jupyter-image/issues/42

Thanks! Ok, I'll prepare a PR for py-base/environment.yml and py-base/conda-linux-64.lock. In the meantime, you'll look into updating miniconda or replacing it with minimamba, right?

I'll ask 2i2c what image bumping cycle we're on, and whether they could set it to daily. Daily will be plenty fast at this time.

emiliom commented 1 year ago

For lack of a better place to park this information:

Once an updated image has been generated, we can request an update to the hub via a PR. Here are the instructions from 2i2c:

the image can be updated anytime you want with a PR like https://github.com/2i2c-org/infrastructure/pull/2100. The automatic one can be treated as a backup, and you can see its activity as it just makes PRs like https://github.com/2i2c-org/infrastructure/pull/2070. So you can always make PRs as frequently as you want updating the values and we'll try to merge them asap.

https://github.com/2i2c-org/infrastructure/pull/2233 is my first PR requesting an update to the hub. Basically, I edited the https://github.com/2i2c-org/infrastructure/blob/master/config/clusters/2i2c/ohw.values.yaml file directly on GitHub (since the edits are small). Under profileList, update the image hash at the end of kubespawner_override.image entries (for python or r image) to the latest one.

Listing of latest and past images:

abkfenris commented 1 year ago

Once a new image is built, Github Actions posts a message to the #infrastructure channel with the image id.

emiliom commented 1 year ago

Lesson for the future: it's a good idea to rebuild the images well in advance even when no specific changes have been identified. That's to expose new conflicts among the latest dependency versions which have come up since the last image build, and to give us a comfortable lead time to resolve them before actual requests for package additions come in.