Support hdfs and s3 path as user home.

jupyterhub / yarnspawner

Spawn JupyterHub single user notebook servers in Hadoop/YARN containers.

https://jupyterhub-yarnspawner.readthedocs.io/

BSD 3-Clause "New" or "Revised" License

19 stars 16 forks source link

Support hdfs and s3 path as user home. #9

Closed Seanspt closed 5 years ago

Seanspt commented 5 years ago

Great work to have yarn integrated! We are considering this option to achieve dynamic resource allocation for our jupyter users. It would be a nice feature to have user home hosted on a remote file system rather than on local disk.

jcrist commented 5 years ago

By "Home" I assume you mean where the notebooks are stored? This isn't handled by the spawner but by the contents manager. A few good options are:

HDFS: https://jcrist.github.io/hdfscm/
S3/GCS: https://github.com/danielfrg/s3contents
Postgres: https://github.com/quantopian/pgcontents

I recommend reading the contents-managers section of the jupyterhub-on-hadoop documentation. This discussing how to install and configure these options as part of a larger JupyterHub install.

Seanspt commented 5 years ago

Thanks a lot. We'll check it.