[vault-app] FIDO does not work until U2F is used

[kop316]

I am noticing when I use the precursor as an SSH FIDO key, my laptop will not find the SSH key after boot up and unlocking the PDDB. However, after I use it as a U2F (e.g. to log into gitlab.com), the SSH key will work as expected. This is on Debian Bookworm.

I can reliably reproduce:

1. Boot up precursor, switch to vault app, unlock PDDB containing SSH resdient key
2. try git clone git@github.com:betrusted-io/xous-core, this will fail:

   
   chris@SpaceballsTheX2100:~$ git clone git@github.com:betrusted-io/xous-core
   Cloning into 'xous-core'...
   The authenticity of host 'github.com (140.82.112.4)' can't be established.
   ED25519 key fingerprint is SHA256:+DiY3wvvV6TuJJhbpZisF/zLDA0zPMSvHdkr4UvCOqU.
   This key is not known by any other names.
   Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
   Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
   git@github.com: Permission denied (publickey).
   fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

3. Sign into github, make sure to use the security key on the Precursor.
4.  try `git clone git@github.com:betrusted-io/xous-core`, this will succeed!

[tweidinger]

Noticed the same behavior. Did you find any simpler workaround?

sign_and_send_pubkey: signing failed for ED25519-SK "user@user-laptop" from agent: agent refused operation

[kop316]

Noticed the same behavior. Did you find any simpler workaround?

sign_and_send_pubkey: signing failed for ED25519-SK "user@user-laptop" from agent: agent refused operation

I did not. This was the simplist I could find

[bunnie]

I'm trying to reproduce this, but, something isn't quite checking out. Did you setup your ssh agent and add the key to it? When I do that, I am able to get it to work without a U2F login.

Here is my set up and my results.

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

 ssh -V
OpenSSH_8.9p1 Ubuntu-3ubuntu0.1, OpenSSL 3.0.2 15 Mar 2022

1. Create authentication key: ssh-keygen -t ed25519-sk -O resident -f testZ
2. Type my PIN to allow key creation.
3. Confirmed there is a key now on vault that is ssh: / -- (FIDO2) openssh
4. Take testZ.pub and paste it into my authorized SSH keys in my github profile.
5. Reboot both the host computer, and the Precursor for a clean slate
6. Attempt to do a clone without any setup: git clone git@github.com:bunnie/iris-controller.git. As expected I get this:

   
   Cloning into 'iris-controller'...
   git@github.com: Permission denied (publickey).
   fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

7. Confirm that an agent is *not* running: `ssh-add -K` and I get `Could not open a connection to your authentication agent.`
8. Start the agent: `eval ``ssh-agent`` ` (double back ticks is actually just one tick). Host responds with `Agent pid 1863`
9. Try to add the resident key to the agent: `ssh-add -K`. I am prompted for the PIN, and this succeeds: 

ssh-add -K Enter PIN for authenticator: Resident identity added: ED25519-SK SHA256:4OpvsoyHR1P/JbLyrkLo465qX87pZeCZ99ybDsW6t7o

10. Attempt the clone, and it succeeds:

git clone git@github.com:bunnie/iris-controller.git Cloning into 'iris-controller'... remote: Enumerating objects: 44, done. remote: Counting objects: 100% (44/44), done. remote: Compressing objects: 100% (32/32), done. remote: Total 44 (delta 10), reused 35 (delta 7), pack-reused 0 Receiving objects: 100% (44/44), 76.82 KiB | 289.00 KiB/s, done. Resolving deltas: 100% (10/10), done.

I re-tried the process again from a clean boot this time running the agent first, and it works without having to do a U2F login.

I'm not even quite sure how you're getting the Precursor to run without having added the device to the agent -- I think in theory you have to have the PIN stored somewhere to access the key? Maybe what's happening is the U2F is triggering a user presence flag and then when you try to do an SSH login that presence is still recorded and somehow it's skipping the PIN requirement?

But please try running the agent and see if this fixes your issue.

[bunnie]

I will add that I tried running the test again, this time just rebooting the host and not the Precursor. On boot, I noted there was some trouble because the host had some process trying to connect to the Precursor in a way that it didn't expect (this is just observing on the protocol analyzer, I don't know what it was but I saw the connection reset a couple times).

When I tried to start the agent the first time, I get this:

ssh-add -K
Enter PIN for authenticator:
Provider "internal" returned failure -1
Unable to load resident keys: invalid format

However, if I immediately re-run the command, it succeeds as expected:

Enter PIN for authenticator:
Resident identity added: ED25519-SK SHA256:4OpvsoyHR1P/JbLyrkLo465qX87pZeCZ99ybDsW6t7o

So, it's also possible that maybe some other program on the host is mucking with the stack in a way that gets the Precursor U2F handler in a weird state, and you need to clear that. Maybe the U2F login does that, but also, starting the agent does the trick too.

[kop316]

Hmm....well this seems to have gotten worse. I tried deleting the SSH Key to see if maybe I did something wrong, but now I get this:

$ ssh-keygen -t ed25519-sk -O resident
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Key enrollment failed: invalid format

I never get a chance to authorize on the Precursor.

EDIT: Ok, so I had to power off and on the Precursor. Now I get this:

chris@SpaceballsTheX2100:~/dev$ ssh-keygen -t ed25519-sk -O resident
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter PIN for authenticator: 
You may need to touch your authenticator again to authorize key generation.
Enter file in which to save the key (/home/chris/.ssh/id_ed25519_sk): 
/home/chris/.ssh/id_ed25519_sk already exists.
Overwrite (y/n)? y
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/chris/.ssh/id_ed25519_sk
Your public key has been saved in /home/chris/.ssh/id_ed25519_sk.pub
The key fingerprint is:
SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM chris@SpaceballsTheX2100
The key's randomart image is:
+[ED25519-SK 256]-+
|                 |
|                 |
|          o      |
|        .+ o     |
|       .S.=      |
|      .o =E      |
|      +.ooo+.    |
|    . oO=BO=+o   |
|     +o=/&&O...  |
+----[SHA256]-----+
chris@SpaceballsTheX2100:~/dev$ ssh-add -L
The agent has no identities.
chris@SpaceballsTheX2100:~/dev$ ssh-add -K
Enter PIN for authenticator: 
Unable to add key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
chris@SpaceballsTheX2100:~/dev$ ssh-add -K
Enter PIN for authenticator: 
Unable to add key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
chris@SpaceballsTheX2100:~/dev$ ssh-add -K
Enter PIN for authenticator: 
Unable to add key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
chris@SpaceballsTheX2100:~/dev$ ssh-add -K
Enter PIN for authenticator: 
Unable to add key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM

EDIT 2: Ok, this seems to be a workaround as well:

chris@SpaceballsTheX2100:~/dev$ git clone git@github.com:betrusted-io/xous-core
Cloning into 'xous-core'...
Confirm user presence for key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
sign_and_send_pubkey: signing failed for ED25519-SK "/home/chris/.ssh/id_ed25519_sk": device not found
git@github.com: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
chris@SpaceballsTheX2100:~/dev$ ssh-add -K
Enter PIN for authenticator: 
Unable to add key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
chris@SpaceballsTheX2100:~/dev$ git clone git@github.com:betrusted-io/xous-core
Cloning into 'xous-core'...
Confirm user presence for key ED25519-SK SHA256:pasm/5BdaeG+tS0Aus/8KZ7w07p9uBt6J8rW8tjsaxM
User presence confirmed
remote: Enumerating objects: 50625, done.
remote: Counting objects: 100% (3853/3853), done.
remote: Compressing objects: 100% (1647/1647), done.
^Cceiving objects:  11% (5569/50625)
chris@SpaceballsTheX2100:~/dev$ ssh-add -L
The agent has no identities.

If I do ssh-add -K, I will be able to use the SSH Key.

[bunnie]

OK cool, so so the TL;DR for the workaround is:

eval `ssh-agent`

if your system doesn't start an agent by default,

and then:

ssh-add -K

And yes, it does look like some turds from the host are confusing the USB stack a bit. When you get the invalid format error, my guess is there is an extra packet stuck in the Rx buffer or something somewhere that didn't get retired, so a plug/unplug or reboot of the device can work around that.

If you happen to bump into the invalid format error regularly, please post the method you use to reproduce it in a new issue. thanks!

[bunnie]

I'll close this issue since I think it's resolved, but if there is more to do please feel free to re-open it.