search

pangeo-data / pangeo-cloud-federation

Deployment automation for Pangeo JupyterHubs on AWS, Google, and Azure
https://pangeo.io/cloud.html
59 stars 32 forks source link

Deploy SSH and SFTP to PANGEO hubs #806

Open yuvipanda opened 3 years ago

yuvipanda commented 3 years ago

Vague conversations with many folks indicated that there has always been interest in some way to transfer large amounts of files efficiently in and out of home directories in JupyterHub. SFTP / SCP has been mentioned as a popular candidate....

I've spent the last 10 days or so working on jupyterhub-ssh which enables some of these use cases. Current features are:

  1. Interactive SSH works perfectly fine - so you can ssh in, run bash, emacs, vim, top, etc.
  2. Environment and home directories would be the same as if you were running this from JupyterHub. So it's an SSH interface to JupyterHub.
  3. SFTP works, so you can copy large number of files in/out fairly efficiently. You can use standard commandline tools, or special GUI programs for this. You can also use something like sshfs to mount the contents of your JupyterHub home directory locally to copy files in / out easily.

I've it experimentally deployed in the JupyterHubs run for UC Berkeley - docs here (https://docs.datahub.berkeley.edu/en/latest/users/ssh.html), and ping me privately for test access to our workshop hub if you wanna try it. github.com/berkeley-dsep-infra/datahub/ is the hubploy-setup repo that also have jupyterhub-ssh deployed...

I think this would be a great fit for the PANGEO hubs. Anyone wanna help me get this added here? :)

TomAugspurger commented 3 years ago

Let me know how I can help out.

I think our CI/CD is a bit fragmented right now. The AWS hub is via GitHub actions and the GCP hub is deployed through .circleci/config.

jbusecke commented 3 years ago

I think this would be a great feature to have. I would be particularly interested if this would enable me to edit files remotely via vscode? This would enable me to fully transition most of my work to the pangeo hubs 🤗

yuvipanda commented 3 years ago

@jbrasch unfortunately vscode would require https://github.com/yuvipanda/jupyterhub-ssh/issues/26 to be implemented before it can work.

jbusecke commented 3 years ago

That makes sense. Thanks for the info, and also many thanks for working on this!