search

NCAR / jupyterhub-deploy

K3d Deployment of NCAR Jupyterhub. This deployment is inspired by and adapted from NERSC JupyterHub Deployment.
Apache License 2.0
2 stars 1 forks source link

Which JupyterHub authenticator should we use? #1

Open andersy005 opened 4 years ago

andersy005 commented 4 years ago

1) NERSC's JupyterHub appears to be using a custom SSH-API based authenticator residing at https://github.com/NERSC/sshapiauthenticator. Per repo's README:

> SSHAuth Authenticator acquires a private SSH key from a SSH Auth API service.

Is this a viable option for our use case?

  1. Currently, users access Cheyenne and Casper with SSH via LDA authentication. Is https://github.com/jupyterhub/ldapauthenticator the right choice for an authenticator? Is there an LDAP server we can authenticate against? If so, what is the process for getting an LDAP client set up on vrealize?

  2. The existing JupyterHub (jupyterhub.ucar.edu) uses the default PAM authenticator via "sshd" pam service

    pam_auth_service = u"sshd"
c.PAMAuthenticator.service = pam_auth_service
c.PAMAuthenticator.open_sessions = False

So far, it is unclear to me which of the three options is appropriate for our current deployment running on vrealize...

As per conversation with @kmpaul this afternoon, we may be able to use a custom SSH authenticator that is based on Fabric(http://www.fabfile.org/).

andersy005 commented 4 years ago

As an update, I was able to fix the kubernetes cluster DNS issue and can confirm that the sshauntenticator works.

kmpaul commented 4 years ago

That's great, @andersy005! So, let's make sure we documentation the pre-requisites for the sshauthenticator to work and then move on to the next step.

To really close this out, we need an answer to questions (2) and (3) above. It seems to me that the answer to (2) above comes from our meeting yesterday:

  1. Currently, users access Cheyenne and Casper with SSH via LDA authentication. Is https://github.com/jupyterhub/ldapauthenticator the right choice for an authenticator? Is there an LDAP server we can authenticate against? If so, what is the process for getting an LDAP client set up on vrealize?

Actually, they appear to access Cheyenne and Casper via ADFS. If we need this for any of our services, like the JupyterHub, then we can get help from EITO to set this up on our VMs. (But setup will be like LDAP.)

  1. The existing JupyterHub (jupyterhub.ucar.edu) uses the default PAM authenticator via "sshd" pam service 
    pam_auth_service = u"sshd"
c.PAMAuthenticator.service = pam_auth_service
c.PAMAuthenticator.open_sessions = False

And it seems to me that the answer to this question is that PAM RADIUS can be set up according to the NCAR/UCAR Knowledge Base:

Does that close out this issue?

What's next?

andersy005 commented 4 years ago

What's next?

I am going to start working on the spawners. By the way, I've opened a new issue (#2) to keep track of the TODOs.

Now that the sshauntenticator is working, I personally think that neither the LDAP nor PAM RADIUS are needed. So, this issue can be closed out.

kmpaul commented 4 years ago

Happy to close this out if you make me (or @NCAR/xdev) collaborators on this repo.

andersy005 commented 4 years ago

@kmpaul, I think this issue is still relevant...

It appears that our current sshauthenticator lacks some features that are crucial for the spawner to work properly. With our current setup, we are not able to launch jupyter notebook servers on the remote host without asking the user to provide necessary credentials again (as far as I know, cheyenne/casper don't support SSH-key based authentication. I tested this today and I still get the password prompt).

I am now more convinced we may want to take full advantage of NERSC's sshapiauthenticator. Using NERSC's sshapiauthenticator in conjunction with sshspawner will make the integration of the spawner and authenticator easy.

The sshapiauthenticator is based on NERSC-MFA-SSHProxy. It's my understanding that when a user authenticates themselves, the following happens:

  1. A certificate/key pair is generated via the SSH Proxy server
  2. The authenticator then acquires the key and the certificate the SSH Auth API service.( Per the NERSC-MFA-SSH proxy slides, the key and certificate are valid for 24 hours)

The spawner uses the retrieved key and certificate to authenticate and launch jupyter servers on the remote host without the user needing to provide credentials again and again.

The sshapiaunthenticator and the sshspawner repos don't have documentations. I had to go through the codebases to figure how pieces fit together. So, I may have missed something and misunderstood some pieces :). The relevant files are here and here and here.

@jbaksta, when you get a chance, we could use your input/feedback in regards to how we should go about authenticating users and launching jupyter notebook servers on Casper and Cheyenne from a JupyterHub running in a VM on EITO's Vrealize platform. I'm particularly interested in learning whether there's an approved and secure way we can use SSH key-based authentication to connect to Cheyenne and Casper.

kmpaul commented 4 years ago

I guess I'm not surprised. My thought would be to just try authenticating with SSH keys. Just try it. There is documentation on the NCAR Cheyenne website that says SSH keys will work on some systems, but it is not clear which systems. So, just try it. See if you can get it to work.

andersy005 commented 4 years ago

just try it. There is documentation on the NCAR Cheyenne website that says SSH keys will work on some systems, but it is not clear which systems.

If you are referring to this documentation page: https://www2.cisl.ucar.edu/user-support/installing-ssh-keys, I followed the instructions without success. I still get the password + 2FA prompt. Could it be that the SSH keys authentication was disabled at some point after that page was published (not sure when that page was last updated)?

kmpaul commented 4 years ago

Yes. It's possible that page is outdated. I'll try it myself.

jbaksta commented 4 years ago

SSH key-based authentication is not allowed from outside the system, but only within the system. A system that would want SSH keys would need to be blessed by the HPC group and most likely in control by the HPC group to react to exploits.

There is not an approved way of doing things outside of the systems or as a third-party platform for interactive sessions. That pretty much would potentially have the ability to sidestep all security elements that we put in place with almost zero traceability on our end. The systems that submit jobs to the queues today are based on a single fixed process and run as a single user or is run through our own cron mechanisms; they're not interactive sessions.

We presently use the PAMAuthenticator because it's very easy for us to integrate it into the system as is and PAM is what really makes the MFA possible on Cheyenne right now with the way the user databases are on the systems. As you noted, it uses the same SSH PAM stack for authentication. There is no intermediate LDAP system that we rely on for authentication; our databases are created on the systems and only rely on RADIUS to authenticate users to the organization. The user database orchestration is probably more complex than what I'd want to give out here.

kmpaul commented 4 years ago

@jbaksta: So, based on what you've said, we are back to square one. Once again, we don't really have a deployment environment we can test with. How do you propose we move forward?

jbaksta commented 4 years ago

I don't know if I would say that. Do you really need the hub portion or would launching your own single server via your own environment work behind a development-driven hub where auth and job launch are still under admin control? Obviously, there would need to be some matching packages, like batchspawner would be required to launch via the job scheduler because it makes a post back to the Hub to create the proxy connection.

It's really not so much the authentication aspect here. It's the step after that that needs more traceability. Presently, it requires access to some sudo which obviously done on your end to facilitate the sudo -> ssh -> qsub/sbatch flow. I do have my issues with SSH keys in user apps like this because it's likely that at least one key will end up in some git repo somewhere; I've seen it too many times to be comfortable. The next thing is that if we were to allow a remote system to do this, how can I have coverage if such system is compromised? How often and whom is auditing that systems logs (not just the hub logs, all the logs).

kmpaul commented 4 years ago

@jbaksta: Thanks for the feedback. You ask some interesting questions, and I'm probably not the right person to answer them.

@andersy005: Can you answer @jbaksta's question above?

Do you really need the hub portion or would launching your own single server via your own environment work behind a development-driven hub where auth and job launch are still under admin control?

kmpaul commented 4 years ago

...I guess the one question I would have is: What would a "development-driven hub" look like? Would someone like @andersy005 be able to make modifications to things like the spawner? Upgrade and reboot the hub? Develop and deploy "satellite" services (like a nbviewer or a BinderHub) that use the JupyterHub for things like auth and job launch? And make any necessary changes to the JupyterHub to make those services possible?