search

pangeo-data / pangeo-cloud-federation

Deployment automation for Pangeo JupyterHubs on AWS, Google, and Azure
https://pangeo.io/cloud.html
58 stars 32 forks source link

nfs home directory mounting on notebook and dask pods #234

Open scottyhq opened 5 years ago

scottyhq commented 5 years ago

We currently have a 'hack' solution for mounting user home directories on efs (see https://github.com/pangeo-data/pangeo-cloud-federation/blob/8c14dd7a06147ee03c1432b01d457c84590dba05/deployments/nasa/config/common.yaml#L5) which differs a bit from the zero2jupyterhub docs: https://zero-to-jupyterhub.readthedocs.io/en/latest/amazon/efs_storage.html

what is the current best practice for mounting efs home directories?

this is also relevant b/c we'd like users to be "bring their own conda environment.yml" to our deployed image that would work with dask KubeCluster. it seems there are quite a few github issues out there and i’m not clear on if that is possible

quoting @yuvipanda "...one way to do it is to share $HOME between workers and your notebook pod. That way, this turns into 'have local conda enviornments'. IMO, I like this more than having conda run on each worker forever to update environment"

so the other thing to sort out is how to get the shared $HOME into our dask_config.yaml https://github.com/pangeo-data/pangeo-cloud-federation/blob/staging/deployments/nasa/image/binder/dask_config.yaml

jhamman commented 5 years ago

I think the Met Office folks (cc @jacobtomlinson) are doing this already, or maybe they have just tried this. Perhaps they can explain their approach.

alando46 commented 5 years ago

@scottyhq it looks like you might have accidentally pasted two of the same link. Curious to follow along - I recently set up EFS with an AWS Batch process (so technically mounting to ECS rather than a generic k8s cluster). EFS works great for our use case (auxiliary data to be used in Sen2Cor processing pipeline), but is it fast enough to host libraries needed for computation for dask workers? At any rate, interested to see what you discover.

jacobtomlinson commented 5 years ago

Yes our new cluster is using EFS as the primary PV storage.

We are using the EFS Provisioner for kubernetes. Our cluster build scripts are OSS so feel free to have a look.

scottyhq commented 5 years ago

just some points of clarification

1) The zero2jupyterhub docs linked above use CHOWN_HOME: 'yes' which is specific to jupyter/docker-stacks: https://github.com/jupyter/docker-stacks/blob/185a9f74b2965ba65ef18f46b5d8494fad445439/base-notebook/start.sh#L52

2) efs-provisioner dynamically creates user home directories on the efs drive. a nice feature is that each user can get their own gid:

For example, we get nasa-staging/claim-scottyhq and nasa-prod/claim-scottyhq, which show up on our efs drive as:

drwxrws--x 8 root 40003 6.0K Apr 27 02:58 claim-scottyhq-pvc-1bb58b6f-67b8-11e9-aa8d-024800b3463a
drwxrws--x 5 root 40000 6.0K Apr 27 03:49 claim-scottyhq-pvc-6cb57249-689f-11e9-93cb-1639dbec9e42

BUT, currently it seems like these folders can't be shared between namespaces. Which doesn't work with our shared home directory with staging and prod deployments. Also, i'm not sure on getting dask workers to point to these dynamically created folders.

So for now we are sticking with the current 'static' nfs solution with the volume-mount-hack referenced in the first comment. but if someone wants to figure out a cleaner way to do this it would be welcome!

scottyhq commented 5 years ago

back to the original purpose of this issue, i think this is now solved w/ our current setup. see: https://github.com/pangeo-data/pangeo-cloud-federation/issues/254#issuecomment-488354689