Switch to 2FA for SSH

[alanbchristie]

From 13th Feb Diamond require SSH connections to use 2FA.

We use SSH connections from the STack (and Worker) Pods in order to access the ISPyB database. A copy of the notice about 2FA follows: -

MULTI FACTOR AUTHENTICATION FOR SSH SESSIONS

As part of Diamond’s ongoing programme to defend against cyber security threats, the Scientific Computing team will implement Multi Factor Authentication (MFA) for SSH sessions used by Diamond staff. Secure Shell (SSH) is a network protocol that gives users a secure way to access computer resources. If you use SSH, please continue reading this email as there will be important changes.

On Tuesday 13th February Diamond will implement SSH MFA on the SSH bastion host - ssh.diamond.ac.uk. Diamond staff using the service will be prompted to access through Microsoft MFA, using the authentication method that they will have previously set up for Microsoft 365. The following Confluence page describes scenarios for the new login experience, depending on their chosen authentication method: https://confluence.diamond.ac.uk/display/DSHK/SSH+MFA

Please be aware of the following consequences:

• If not already done, you must set up Microsoft MFA as per instructions on SharePoint;
• You will be asked to go through the MFA flow on every SSH login to the ssh bastion hosts;
• The MFA will apply to all SSH-based connections to the host (SSH, SFTP, SCP, sshuttle and rsync over SSH);
• If you use the SSH service as part of your science workflow, please contact scientific computing, urgently, by creating a ticket at https://schelpdesk.diamond.ac.uk

If you have queries or concerns regarding the implementation of MFA for SSH then please contact: SCHelpdesk@diamond.ac.uk.

Thank you,

Scientific Computing Team

[alanbchristie]

The Fragalysis stack uses sshtunnel and this should allow both username/password and username/private-key connections (see Example 2 in its documentation).

If we need to switch to a private key it will need a minor code change in the stack but also the delivery of the private key via a ConfigMap that is mapped as a file into the Stack (and worker) Pods.

[Waztom]

To minimize code changes needed - have asked SC to provide a PW for the account.

[Waztom]

@alanbchristie SC have requested we use a public SSH key.

The account does not have an associated PW, from SC :

We would very much prefer to use a key as it makes it easy to restrict which networks the account is used from. If a key is really impossible to use, please let us know ASAP.

[Waztom]

@alanbchristie so turns out the MFA SSH is going to be implemented tomorrow (I found out this evening). Can you please let me know soonest if a key is not going to be possible to use/implement within the week (See SC comment above)? Have moved this back to Purple release - to be discussed at next meeting.

[alanbchristie]

The sshtunnel package does appear to offer a private key mechanism. This should be a simple change (affecting code and deployment with a new ConfigMap to hold the key). I can test from a shell in the stack Pod.

I'm away from my desk atm - but will check when I'm back in the office this morning.

[alanbchristie]

For clarity let's publish the account username and public SSH key in this issue.

[alanbchristie]

The public key for the account is:-

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDqTrmu9MFMMBzmGesIofQ2mTfVhpWpmT8VvrrPbMZHiAWULGFhrzQvU0ywSa66ZjvZAcKVGmScivRwM0ViKoELp7BinCINAQ51fHlaJgwmCi/sYWvO11yNuyA2PvPv3gdPYVGVC4qtURJuLCeiv49lvijX16mKZSu3RIhMXBoRelYDmVhI0A1Ejvl8VmqD8kaKxcJWI/yq5B1UctXklM7vVbtwRmbdu+3JyliU4Pv5lIsdr3JlB11pKdMK1gmk/JO5NPlrJUgf6naCVe/JXkst3b7XYwICJL0LDZ4p2Z4Wos/u2lON1lVeXFv9zE0QBYnCcpPDK3RyLhftcZX/BMYuUyp9zQmaaXQHkE50zTxcY1MxdgHA3RAKJz7EvxRiN9duc3wbhj/0wjlZ64HL8lM/nMNnEPMefya5ANEyP1DmJEsK+0oNIQ2OboAlgMIy4AET93JKf6v1n6RYWagUIC+NKQKZX3HHTVoIi9Yw8Oa6Harn6X0iMSvc4RVFmzt+go0= abc@Dallas

The account username is: -

fragalysis-stack

[alanbchristie]

ATM I'm getting this when trying to establish a key-based SSH tunnel (the password approach works): -

sshtunnel.BaseSSHTunnelForwarderError: Could not establish session to SSH gateway

I am seeking help via the ticket (SCHD-3366)

[Waztom]

@Waztom will confirm that the public key is in fact linked to the functional account and request a PW for the account. @alanbchristie confirms that SSH public key is not associated with the functional account.

[Waztom]

@alanbchristie has confirmed that we require a PW to be able to SSH via locations/deployments other than the STFC cluster. @Waztom will follow up with SC to motivate getting a PW.

[alanbchristie]

The backend (in version 2024.02.1) now supports password and private key-based SSH tunnels. To use keys the user will need to use playbooks from the https://github.com/xchem/fragalysis-stack-kubernetes repository (on branch m2ms-1316), which have not been released. With the latest backend (and old playbooks) stacks will continue to use SSH passwords.

Issues with private-key SSH tunnels to the MySQL database prevent the use of SSH private keys (for now).

Once the playbooks have been merged to the main branch private keys should become the default. Individual AWX Job Template variables will need to be inspected to ensure that no over-ride of stack_ssh_* or stack_isbyb_* variables are in place.

[alanbchristie]

The changes to switch SSH connection from password to private key file seem to work with the latest backend ... but ... when using the key-file I find that I'm unable to connect to the ISPyB MySQL database through the established SSH tunnel. I get an Administratively prohibited ERROR, even though the same MySQL credentials are used regardless of whether it's a password or private-key based tunnel.

In summary, password SSH works, private-key SSH DOES NOT.

To use private-key SSH tunnels we will need the changes in the Kubernetes repository (which have not been merged yet). Until we fix key-based tunnels these changes will need to wait.

Also ... remember that automated (unattended) access to the SSH server from outside the Diamond network is now not possible.

[Waztom]

@Waztom will talk to SC/James about IP address range issue - M2M guys have IP addresses that change - can we have an account to key without IP range enforced?

[alanbchristie]

Contacts for the original SSH connection issue are: -

• Alan Huggan (STFC) helped on the problem under STFCCLOUD-5352. He suggesting the problem was with Diamond and that they need to allow 192.168.253.* addresses and 130.246.212.100.

• Richard Lear (and Richard Taylor) from Diamond Support then helped with the SSH connection issue. That was done under SCHD-3210 and SCHD-3366. I think the account was originally "blocked".

Until recently we were able to connect using account and password and the private key file prevented us from connecting to the ISPyB database.

[alanbchristie]

We are told that Frank's account (the one used to create the SSH Tunnel) will be switched to 2FA soon (tomorrow?). This will cause the stack (even the legacy production stack) to fail to authorise anything. In summary: -

1. We have a service account (fragalysis-stack) that has a private key file. We can create an SSH tunnel with this, but cannot connect to MySQL (The ISPyB database).
2. We have a user account (Frank's) without a private key (i.ee just a password). We are able to create an SSH tunnel with this AND connect to MySQL.

At the moment...

If we switch Frank's account off all authenticated access (to all stacks) will be lost.

WE URGENTLY...

...need to be able to continue use the user account or someone has to explain why access to MySQL is unavailable through the service-account-created tunnel.

[alanbchristie]

Tests today seem to indicate that the private-key approach to creating an SSH tunnel is working. This relies on a branch in the kubernetes-fragalysis-stack repository (m2ms-1316), wand this can now be merged into the main branch, tagged, and used.