authentication error: using ssh CLI to access controls hosts

[crispd]

Authentication error: using ssh CLI to access controls hosts

just the gateway, for now

Behavior:

Getting authentication error when trying to use the ssh CLI from a windows terminal shell (PowerShell) to access one of the controls gateways. I can get access using the Controls PuTTy launcher, and even from the terminal when I use plink <controls_gateway>, but whenever I try to use ssh <controls_gateway> the GSS_S_FAILURE error is thrown and ultimately 'permission denied'

To Reproduce:

Replacing bracketed values with strings that suit the enclosed description (see below comment for additional reference)

1. Use an AD managed windows laptop with openssh and updated version of 'Accelerator Controls Customizations'
2. Setup an ~/.ssh/config file with the following:

   Host <controls_gateway>
   HostName <controls_gateway_TLD>
   User <my_username>
   GSSAPIDelegateCredentials yes
   GSSAPIAuthentication yes
   ForwardAgent yes
   ForewardX11Trusted yes
   ForwardX11 yes

3. Run this line for auto starting an ssh agent Set-Service ssh-agent -StartupType Automatic (for autostarting in future) then Start-Service ssh-agent (my ssh config has github specific host definitions that I use id_ed25519 keys for. the agent adds them on startup for me, I don't know if this is relevant to my issue)
4. Run ssh <controls_gateway>
5. Based on the output:
   • I get as far as the 'NOTICE TO USERS' bit that says 'This is a Federal computer...',
   • gssapi-with-mic authentication is attempted
   • GSS_S_FAILURE is thrown, and ultimately 'my_username>@<controls_gateway_TLD Permission denied (gssapi-with-mic)'.

Extra Context:

The output of klist indicates that my cache isn't updating like I expect it to. It will update sometimes, but not every time. For instance, in one test, I saved the klist > klist.log then destroyed credentials, then renewed, and the klist output didn't indicate any change.

---

Affected system:

• [X] Console

[crispd]

Resolution:

The best solution, for windows users what wish to use SSH, appears to be to use SSH commands from a WSL instance. (Keeping in mind, of course, that the plink PuTTy command is an option for properly setup users as well).

I ended up choosing to separate my config definitions into new files ~/.ssh/config.fermi & ~/.ssh/config.git, then using these lines in ~/.ssh/config to join them:

Include ~/.ssh/config.git
Include ~/.ssh/config.fermi

Then I added shell functions that manage my ssh-agent (seen in this gist), and ran the dnf command(s) seen here. Thanks to Tim Z. for this solution!

...and now commands like ssh <host_alias>, ran from WSL2 (AlmaLinux 9.2), seem to be working as expected.

Some details on Windows' built-in openSSH package:

This stackoverflow answer is the best description I've found as to the difficulty of using windows' built-in openSSH to handle GSSAPI: https://stackoverflow.com/a/75474673

There is probably a way to get ssh to handle GSSAPI in windows, but once it became clear just how easy it was to make WSL work (a simple one-liner to install packages from fermilab's internal rpm mirror), it no longer made sense to spend the extra time figuring it out. Simply upgrading to the OpenSSH Beta for windows proved insufficient (I believe this build includes handling of SSPI, but this isn't quite the same, so additional work would be needed).

Extras

In going through this, I discovered a shell function which may be helpful to those trying to troubleshoot their own, possibly complex, ssh configs. If curious, refer to the gist here.