more deployment updates

[bolliger32]

Workflow

• [ ] Closes issue #
• [x] Notebook:worker pairing builds & passes local build, visual inspection & client.map tests
• [x] Passes travis tests
• [x] Image deployed on test2.climate-kube.com
• [x] Notebook+worker passes test-hub deployment

Summary

Trying to pick up where @delgadom left off in #102 , and also integrate a few things from #91.

In #102, we currently had the following problems. The check ones should be fixed from this PR. The unchecked ones are not, b/c I'm not quite sure how to do them and/or I'm not sure if we need to do them before these changes get merged and deployed to compute.rhg:

• [x] workers don't seem to spin down
• [x] mysterious "worker already exists" error
• [x] RuntimeError: incompatible bytecode version encountered when using datashader on a dask cluster
• [x] autocomplete is populated with everything in globals()
• [x] the dask.distributed progress bar doesn't seem to pop up
• [x] Make sure geopandas is still vectorized. To solve the packages in this release, I switched from the conda-forge/label/dev::geopandas to simply conda-forge::geopandas. The label/dev version was the recommended method for installing the cythonized version of geopandas, but I am honestly not sure what is anymore. If this is not vectorized, this will render geopandas unusable for many of our applications. (this PR installs the conda-forge/label/dev version, which is quite old but still the only feasible way I can find to install the geopandas-cython branch. I believe the newest location for this cythonized python is at https://github.com/geopandas/geopandas/pull/1030 but the current commit does not build. Hopefully this will soon be merged back into master and we can go back to the conda-forge main repo.)
• [x] making sure we have the same environments in notebook and worker images (#95) (this PR addresses this by using a common environment.yml file for the base packages, and then adds to it for the notebook image).
• [x] Check to make sure modifying the environment file modifies the docker image and hash when pulling from a cached dev container with no other changes. (assuming this is referring to the travis build? I'm not sure the cached dev container helps too much for the build and in fact slows things down if you've made a bunch of upstream changes in the docker process b/c it pulls this large image and then starts over. So I got rid of it for now and figure if you're developing on dev-* or whatever branch you want, you can add it back in)
• [x] pin package versions so we can ensure this image can be built in the future
• [ ] maybe pin helm chart dependencies rather than using inequalities? (What are the helm chart dependencies in the docker_images repo?)
• [ ] Make sure we can upgrade the cluster without deleting user directories or anything else important. The helm chart updates have some pretty significant breaking changes, and involve force-upgrading the cluster and deleting some services. See jupyterhub/zero-to-jupyterhub-k8s#638
• [ ] (Down the line) Once this is merged & deployed on all servers, we should file an issue to update our jupyterlab templates (see RhodiumGroup/rhodium-custom-jupyterhub-templates#1). This would break the current deployment since the template repo structure has changed significantly since the version of jupyterhub currently deployed. If we can figure out how to deploy templates from a templates repo branch in the helm chart we could test changes before deployment.

Supersedes #101 (where we just changed the name of the google credentials environment variable). Closes #95

Features

In general, I did a few things:

1. Rearrange the structure of the build process so that we remove some duplication in the notebook and worker dockerfiles
2. Bypass the intermediate netcdf images for both notebook and worker (as in #91)
3. Switch to using the same Ubuntu base image for worker and notebook (as in #91, not sure this is 100% necessary but back when it was giving us slightly different netcdf versions b/c the repos for the different distributions was different)
4. Unpin every python package, let conda figure out the best spec, and then re-pin everything
5. Dropped the worker env on workers. Was there a reason we were doing that? It was just frequently coming up as a hassle, but not an insurmountable one if there was a reason we wanted a worker environment that we were always inside of, rather than just using the base env

[bolliger32]

Right now, we're not initializing gcloud so that users can perform gsutil commands. We could add a simple line to the notebook/prepare.sh like we do for worker/prepare.sh but I didn't know if we actually want to do that or not, so I left it out for now. I also think we might be able to drop some of the optional environment handling in notebook/prepare.sh? Like, are we going to pass an environment.yml file to the notebook somehow? and/or are we going to pass extra_conda_packages or extra_pip_packages? If not, we could just drop that stuff for clarity. But it's not doing any harm staying in there.

[delgadom]

Back from the honeymoon. This looks awesome! Thanks so much.

Re your last comment - I think we can get rid of all of this in prepare.sh - I don't think there's any way that these would be used so delete away! Also, I'm a fan of not initializing gcloud in prepare.sh, since the user can set up different auth and store it in the home directory (e.g. with gcloud init) and we wouldn't want to manually override that with rhg-data credentials or something.

The helm chart milestones all pertain to simultaneous development tasks in the helm-chart repo, so aren't necessary for this PR - we just need to keep an eye on them.

Have you been able to deploy this? Which server? Excited to give this a spin :)

[bolliger32]

@delgadom right on! Hope you got a good tan :) this should be up on both compute-test and test2. Give it a whirl! See if you can break something :)

[bolliger32]

FYI - looks like geopandas is abandoning their cython branch in favor of integrating a new package called pygeos (geopandas/geopandas#1155). But not quite ready yet so will stick with the dev version for now