Bundled version of OpenSSH with macOS Monterey doesn't support FIDO2 yubikeys

[stoggi]

With the latest version of macOS Monterey (12.0.1) Apple have bundled a newer version of OpenSSH (OpenSSH_8.6p1, LibreSSL 2.8.3) but seem to have compiled it without --with-security-key-builtin. This is disappointing, but makes sense, as it would be unlikely that Apple would redistribute libfido2. However even though the bundled version man pages indicate that it should support creating ecdsa-sk and ed25519-sk key types it doesn't work.

/usr/bin/ssh-keygen -t ed25519-sk -f ~/.ssh/test_key_id_ed25519_sk
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Key enrollment failed: unknown or unsupported key type

I saw that in version 1.3.0 libfido2 used to compile the helper library sk-libfido2 which would connect OpenSSH to a Yubikey by specifying SSH_SK_PROVIDER or passing it as a command line parameter to ssh-add, ssh-keygen, or ssh. This was removed in 1.3.1 as it was picked up by the OpenSSH codebase:

https://github.com/Yubico/libfido2/blob/e1c761a9a03ffe95fbb11367252a0d091ea6a1d5/NEWS#L127

So, I'm proposing that we add the helper library back in to libfido2, so that Yubikey users can create ecdsa-sk and ed25519-sk with yubikeys using the bundled versions of OpenSSH in macOS. That way a user can do:

brew install libfido2
ssh-keygen -t ed25519-sk -w /usr/local/lib/sk-libfido2.dylib -f ~/.ssh/test_key_id_ed25519_sk
ssh-add -S /usr/local/lib/sk-libfido2.dylib
ssh -O SecurityKeyProvider=/usr/local/lib/sk-libfido2.dylib user@host

Why not just use brew install openssh? Because the ssh-agent that comes bundled with macOS has some useful integrations with keychain, and launchctl. As well as the expectation that the bundled SSH version 8.6 should work with a Yubikey out of the box.

I am willing to contribute this change if there is interest for it. I think it has value in libfido2 rather than a separate github repository.

[martelletto]

Sounds good to me. Happy to help, if needed.

[stoggi]

I've had a look into it, and tried reintroducing the library from 1.3.0 including upgrading the API to work with newer versions of OpenSSH.

However, I always encountered the same error:

Key enrollment failed: unknown or unsupported key type

Where I could see the openssh-portable version picking up my compiled library. I suspect that instead Apple may have actually compiled it with --disable-security-key, which would cause sshsk_enroll and sshsk_sign to return without attempting to load the dynamic library:

https://github.com/openssh/openssh-portable/blob/e86968280e358e62649d268d41f698d64d0dc9fa/ssh-sk-client.c#L314-L316

Sad. Most likely this is something we'd need to complain to Apple about, as I can't see any way around it if this is the case.

[martelletto]

I had a quick look and that was my impression as well. Bummer. Thanks for taking the initiative, though. Much appreciated!

[cap10morgan]

I filed a feature request for this here: https://www.apple.com/feedback/macos.html

If enough of us do it, perhaps it could appear in a future release of macOS :)

[martelletto]

@cap10morgan, do you happen to have a radar number or other tracking information for the feature request you have opened? Thank you!

[cap10morgan]

@martelletto No I don't think that form provided anything like that. Or if it did I didn't record it. I think submitting a bunch of duplicate requests is all we've got there, but I'd be happy to be corrected.

[martelletto]

@cap10morgan I see. Thank you!

In the understanding that linking to duplicates helps triage, would anyone have a radar number I could link to?

[timfallmk]

In the spirit of trying to get this moving, I've submitted the following, which at least gives us a tracking number. I assume more reports referencing it would get it more attention. Note the Recent Similar Reports: None marker.

[SebDominguez]

Wondering why the issue is closed but I still can't use FIDO2 to secure my SSH connection.

All suport documentation from Yubiko doesn't indicate clearly that macOS currently is incompatible with FIDO2 giving the current shipped openSSH build.

Please considering disclaiming this gotcha clearly since I was mislead reading all the developers.yubico.com docs and bought many keys just to find out that I can't use them as advertised.

[martelletto]

Use of FIDO2 security keys is not possible with Apple's SSH because Apple explicitly disabled the feature. That said, use of FIDO2 security keys with SSH is possible on macOS through Homebrew or if you build OpenSSH yourself. The issue is closed because there is nothing we can do from our side. I will make sure to edit https://developers.yubico.com/SSH/ and https://developers.yubico.com/SSH/Securing_SSH_with_FIDO2.html to highlight the fact that usage with Apple's SSH is still not possible. Happy to help if you have further questions.

Edit: emphasis

[SebDominguez]

Thanks for your help, the homebrew version doesn't seems to support -O resident -O verify-required options since when I try to set any of those I have an Key enrollment failed: invalid format error.

Also if I use the homebrew version then the UseKeychain options on my .ssh/config trow me errors.

Is there a way to build openSSH with all the macOS bells and whistles + support for FIDO2 ?

[martelletto]

You could try to build https://github.com/apple-oss-distributions/OpenSSH without https://github.com/apple-oss-distributions/OpenSSH/blob/main/configure-for-osx.sh#L7 (which is what https://github.com/apple-oss-distributions/OpenSSH/pull/1/ does).

Regarding Homebrew, does your security key have a PIN set? If not, you can use fido2-token -L to list keys, and fido2-token -S <device path> to set a PIN. ssh-keygen -O resident -O verify-required should work then.

[SebDominguez]

I've setup a PIN using the YubiKey Manager software, and it seems to me that the setup was successful since I could login into macOS using the key+PIN. I'll try using the CLI. Only thing that I didn't set was the management key but I don't think that was necessary.

Edit: Ok just find out that PIV PIN != FIDO PIN. Still learning

[georgetasioulis]

Just got my first YubiKeys today and wanted to create ed25519-sk keys on macOS Monterey 12.3.1, only to realize that it's not supported; bummer...

Hope to see this supported soon.

[martelletto]

It is supported but disabled in Apple's SSH. If you'd like to use ed25519-sk keys on macOS, install ssh from Homebrew.

[georgetasioulis]

Yeap I've read the rest of the thread - I understand that OpenSSH v8.6 supports it, it's was just not built & bundled by Apple with --with-security-key-builtin. As someone mentioned above, replacing ssh with Homebrew's will break integrations with keychain etc, so that's why I'm not doing it.

PS: Apologies if I sounded like I'm complaining to Yubico - my rant is to Apple :)

[simpsora]

@martelletto apologies for jumping on the thread, but I'm seeing the same thing, and the solutions here are not working for me.

I've got 2 yubikeys:

• YubiKey 5C Nano (5.1.2) [OTP+FIDO+CCID]
• YubiKey 5 NFC (5.1.2) [OTP+FIDO+CCID]

macOS:

• Monterey, 12.3.1

OpenSSH from homebrew:

• OpenSSH_9.0p1, OpenSSL 1.1.1n 15 Mar 2022

libfido2:

• 1.10.0

I've set a FIDO PIN using ykman fido access, and the verify-pin command there accepts my PIN.

Calling fido2-token -S results in the following error:

fido2-token: fido_dev_set_pin: FIDO_ERR_NOT_ALLOWED

Whenever I try to generate a resident key, I get the following:

$ ssh-keygen -t ecdsa-sk  -O resident -O verify-required
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
Enter PIN for authenticator: <enters correct pin>
Key enrollment failed: unknown or unsupported key type

Following along from above, I also tried to build openssh from apple-oss-distributions but ran into lots of problems compiling (most recent is issues with dyld headers).

From my reading these devices should support resident keys, but it's not working. Am I holding it wrong? Any suggestions to get it working would be most welcome :)

[martelletto]

@simpsora, it looks like the key you are using does not support credProtect, which is required for -O resident and -O verify-required to work. credProtect is a FIDO 2.1 extension. You can check it using fido2-token -I <device> | grep credProtect:

$ fido2-token -I /dev/hidraw8 | grep credProtect
extension strings: credProtect, hmac-secret

Edit: clarify that credProtect is needed for -O resident to work.

[simpsora]

@martelletto indeed:

$ fido2-token -I ioreg://4294990454
proto: 0x02
major: 0x05
minor: 0x01
build: 0x02
caps: 0x05 (wink, cbor, msg)
version strings: U2F_V2, FIDO_2_0
extension strings: hmac-secret
aaguid: cb69481e8ff7403993ec0a2729a154a8
options: rk, up, noplat, clientPin
maxmsgsiz: 1200
maxcredcntlst: 0
maxcredlen: 0
fwversion: 0x0
pin protocols: 1
pin retries: 8
uv retries: undefined

I assume that FIDO 2.1 must be supported in later firmware versions for these keys, given that the yubi/github blog post said:

If your security key supports FIDO2 resident keys*, like the YubiKey 5 Series ...

Searching for credProtect specifically, looks like it's part of firmware 5.2.3.

Looks like it's SSH via PIV for me, and time to order some new keys!

Thanks for your quick reply :)

[ghost]

Just thought I'd chime in here.

I've been using Yubkey with an ECCP256 key via the PIV module on both Linux and Windows for the last year or so. It's been a rock solid solution. I recently got myself a Mac Studio and noticed that I couldn't login to any of my servers via SSH. I'd get the following error message:

ssh -I /usr/local/lib/opensc-pkcs11.so user@10.2.2.2 unknown or unsupported key type

I saw a similar issue on the OpenSC Github community and they pointed at OpenSSH as the problem (https://github.com/OpenSC/OpenSC/issues/2559)

I took one of my Yubikeys and re-created a PIV auth certificate with RSA2048 instead of ECCP256 - It works.

i guess I'll just migrate over to RSA2048 for compatibility until a more robust solution comes alone.

I hope we get support for ECCP256 and ECCP384 in the not so distant future.

[timfallmk]

@sourcenix That's actually an entirely different mode than the one being discussed here. Native fido2 support in openssh doesn't use the PIV module. Sad to hear it doesn't work there either.

[thelastlin]

Good news everyone!

I believe this problem maybe have been "fixed" in the latest version of macOS!

tl;dr FIDO-based ssh keys should now work in macOS Ventura (13.1).

ssh-keygen -t ed25519-sk -O resident -O application=ssh:<your-text-here>
ssh-copy-id -i ~/.ssh/id_ed25519_sk <host>
ssh <host> -i ~/.ssh/id_ed25519_sk

Longer Explanation

I recently came back to this randomly and decided it was time to test it again. macOS Ventura (13.1) reports using a bundled version of OpenSSH at OpenSSH_9.1p1, OpenSSL 1.1.1s 1 Nov 2022 which, like the version mentioned in the original post, should support FIDO-based ssh keys, both resident and non-resident.

To my surprise, I got a different result from the ssh-keygen -t ed25519-sk operation which appeared to interact successfully with the YubiKey. After some quick checking I was able to verify that the key was present on disk and that I could also generate a resident key that would show up on the YubiKey itself.

I've only tested this enough to make sure the basic functions (authentication of a remote) work and there do appear to be some gotchas. Here's what I've tested:

ssh-keygen -t ed25519-sk -O resident
<prompt for pin/fingerprint>

ssh-add -K
<prompt for pin/fingerprint>
Unable to add key ED25519-SK SHA256:35ivjStAlI+knCB8mtN699BJkgtItvBhxZId5Bvkx14

ykman fido credentials list
<prompt for pin/fingerprint>
a970c00f...    ssh:<text>                         openssh          openssh

ssh-copy-id -i ~/.ssh/id_ed25519_sk empok-nor
<prompt for pin/fingerprint>
<prompt for presence>
Copied 1 Keys

ssh empok-nor -i ~/.ssh/id_ed25519_sk
<prompt for pin/fingerprint>
Success!

A few things to notice:

1. `ssh-add -K` seems unable to add `sk` keys to the identity agent (I use the 1Password SSH Agent, which may be interfering. I did not check)

2. `ssh-keygen` and `ssh-keygen -K` (for generating and downloading resident keys, respectively) seem to work, but may incorrectly report "invalid format" or similar errors for incorrect pin/fingerprint/auth attempts during the process

3. Configured identity agents (such as 1Password's) will override a specified key, even if given with the `-i` flag. I believe this is the expected behavior.

Obviously I have not extensively tested this, nor have I checked what other versions of macOS/OpenSSH might be supported. I can't find any announcement or documentation mentioning this change, so it might just have been rolled up with a newer bundled version of OpenSSH. I just thought you all might want to know!

Tested with:

* macOS Ventura (13.1)

Device type: YubiKey C Bio - FIDO Edition
Firmware version: 5.5.6
Form factor: Bio (USB-C)

Edit: A grammar

Thanks for this information. It seems that this problem hasn't been fixed with macOS 13.2RC (22D49).

• macOS 13.2 RC (22D49)
• /usr/bin/ssh -V: OpenSSH_9.0p1, LibreSSL 3.3.6
• /usr/bin/ssh-keygen -t ed25519-sk -O resident: unknown key type ed25519-sk, return -1.

[SebDominguez]

Unfortunately this seems to be a Ventura fix only. I'm stuck with Monterey.

[timfallmk]

I may have completely jumped the gun on this one in my excitement. I mixed and matched upstream OpenSSH ssh-keygen (from hombrew) and bundled ssh.

That was a stupid mistake. Now I'm just embarrassed 😞 .

So sorry everyone! (The dream lives on)

Edit: I will let the shame set in for a day and then delete my original comment

[gwire]

Just noting that this is still the case with the binaries in macOS 13.2 (Ventura)

% /usr/bin/ssh -V
OpenSSH_9.0p1, LibreSSL 3.3.6
% ssh-keygen -t ed25519-sk 
unknown key type ed25519-sk
% ssh-keygen -t ecdsa-sk 
unknown key type ecdsa-sk

[javielico]

The issue seems to be persisting in macOS 13.2 (Ventura).

[obelisk]

Appears to be the same in 13.3

[m3dwards]

Getting the same in 13.4 Beta 3

unknown key type ed25519-sk

[obelisk]

This is fixed in macOS Sonoma!!!

But, it requires passing in a compiled SO for the middleware

[obelisk]

My custom FIDO agent seems to be working well without brew openssh, without the middleware.

Git, ssh-add, ssh all seem to work

[thelastlin]

Someone might want to build the middleware as a standalone SO/dylib in macOS Sonoma, here is the guide:

https://gist.github.com/thelastlin/c45b96cf460919e39ab5807b6d20ac2a

[martelletto]

It would probably make sense to have the middleware as a maintained, regularly released project. @LDVG @kongeo is this something Yubico would be interested in maintaining? CC @djmdjm

[gwire]

macOS 14.0

% /usr/bin/ssh -V
OpenSSH_9.3p2, LibreSSL 3.3.6
% ssh-keygen -t ed25519-sk 
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
No FIDO SecurityKeyProvider specified
% ssh-keygen -t ecdsa-sk   
Generating public/private ecdsa-sk key pair.
You may need to touch your authenticator to authorize key generation.
No FIDO SecurityKeyProvider specified
Key enrollment failed: invalid format

oh, that's different.

[vcsjones]

Yep. Seems like OpenSSH on Sonoma can work with Security Keys, if you build the SK provider yourself.

[cavokz]

Yep. Seems like OpenSSH on Sonoma can work with Security Keys, if you build the SK provider yourself.

Fantastic! How can I reproduce your success? It would be a nice post somewhere :)

[MichaelRoosz]

I have packaged the lib with a brew formula:

brew install michaelroosz/ssh/libsk-libfido2

use with: SSH_SK_PROVIDER=/opt/homebrew/lib/libsk-libfido2.dylib

If you want to build it yourself: https://gist.github.com/thelastlin/c45b96cf460919e39ab5807b6d20ac2a

credits to @thelastlin for the instructions on how to build the lib 👍

[mlcsthor]

I have packaged the lib with a brew formula:

brew install michaelroosz/ssh/libsk-libfido2

use with: SSH_SK_PROVIDER=/opt/homebrew/lib/libsk-libfido2.dylib

If you want to build it yourself: https://gist.github.com/thelastlin/c45b96cf460919e39ab5807b6d20ac2a

Hi, I've just tried your lib on a clean installation of Sonoma and I get an 'invalid format' error.

ssh-keygen -t ed25519-sk -O resident -O verify-required

Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
Provider "/opt/homebrew/lib/libsk-libfido2.dylib" dlopen failed: (null)
Key enrollment failed: invalid format

[MichaelRoosz]

@mlcsthor could it be that you have an intel mac? then it could work with: SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib

[mlcsthor]

@MichaelRoosz Indeed I have an Intel mac and changing the variable works. Thanks a lot!

[tdb]

Any luck making this work with the macOS ssh agent?

Could not add identity "id_ed25519_sk": agent refused operation

It worked using the ssh-add from Homebrew, but not the macOS one. Which is a little odd... I was initially assuming the agent needed the SSH_SK_PROVIDER environment too, but I guess not.

For what it's worth, it all worked fine adding a non-sk key to the agent.

[stoggi]

@tdb A buddy of mine (thanks @scottgeary) had the answer to this one.

ssh-agent only loads providers from an allow-listed set of directories, configurable with -P to ssh-agent, see the manpage of ssh-agent:

     -P allowed_providers
             Specify a pattern-list of acceptable paths for PKCS#11 provider and FIDO authenticator
             middleware shared libraries that may be used with the -S or -s options to ssh-add(1).
             Libraries that do not match the pattern list will be refused.  See PATTERNS in ssh_config(5)
             for a description of pattern-list syntax.  The default list is “/usr/lib/*,/usr/local/lib/*”.

I tried this, and as I have an intel mac @MichaelRoosz homebrew script (thank you btw for this!!) puts libsk-libfido2.dylib into one of the default locations, however it is a symbolic link to ../Cellar/libsk-libfido2/9.4p1/lib/libsk-libfido2.dylib, and so is rejected by ssh-agent. If you manually copy libsk-libfido2.dylib into /usr/local/lib then the builtin ssh-add does work:

❯ /usr/bin/ssh-add -S /usr/local/lib/libsk-libfido2.dylib tempkey
Identity added: tempkey

[MichaelRoosz]

Thank you for the feedback @stoggi

To automatically copy the lib to the correct place, I have created this homebrew cask:

brew install michaelroosz/ssh/libsk-libfido2-install

It installs "michaelroosz/ssh/libsk-libfido2" and needs sudo access to copy the file to "/usr/local/lib/libsk-libfido2.dylib"

then you need to use

SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib

for both Intel and Apple M1/M2 silicion to make it work

[tdb]

Thanks @stoggi and @MichaelRoosz, this worked great! Really happy to finally have a working solution using (mostly!) the builtin tools.

[oezh]

Thank you for the feedback @stoggi

To automatically copy the lib to the correct place, I have created this homebrew cask:

brew install michaelroosz/ssh/libsk-libfido2-install

It installs "michaelroosz/ssh/libsk-libfido2" and needs sudo access to copy the file to "/usr/local/lib/libsk-libfido2.dylib"

then you need to use

SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib

for both Intel and Apple M1/M2 silicion to make it work

Did this, but im getting:

ssh-keygen -t ed25519-sk -O resident
Generating public/private ed25519-sk key pair.
You may need to touch your authenticator to authorize key generation.
No FIDO SecurityKeyProvider specified
Key enrollment failed: invalid format

Sonoma, Mac Studio M2

[MichaelRoosz]

@oezh this error happens when "SSH_SK_PROVIDER" is not set or empty.

does it work if your run both in one line?: SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib ssh-keygen -t ed25519-sk -O resident

if yes, you need to permanently set it for your shell

[oezh]

@oezh this error happens when "SSH_SK_PROVIDER" is not set or empty.

does it work if your run both in one line?: SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib ssh-keygen -t ed25519-sk -O resident

if yes, you need to permanently set it for your shell

That worked perfectly. Added to shell.

Thank you @MichaelRoosz

[orolhawion]

You may also set the following line into ~/.ssh/config to support sk keys:

SecurityKeyProvider /usr/local/lib/libsk-libfido2.dylib

Authentication would then work without SSH_SK_PROVIDER. However, SSH_SK_PROVIDER is required for key generation though.

[ulischaefer]

Do I need to setup SSH_ASKPASS in my environment? When I use my Yubikey on linux, I have to type in my PIN twice - once after "ssh-add -K" and once again when opening the ssh connection with the key plugged in. But when I try to use the same setup on Mac OS, I have to type the PIN once after "ssh-add -K" and that's it, but when I open the ssh session, pubkey authentication fails and falls back to keyboard authentication.

[stoggi]

@ulischaefer when you generated your key, it sounds like you specified the verify-required option:

verify-required Indicate that this private key should require user verification for each signature. Not all FIDO authenticators support this option. Currently PIN authentication is the only supported verification method, but other methods may be supported in the future.

And I think you are correct, ssh-agent would use SSH_ASKPASS to request the PIN, as seen in the 8.9 release notes:

• ssh-add(1), ssh-agent(1): allow pin-required FIDO keys to be added to ssh-agent(1). $SSH_ASKPASS will be used to request the PIN at authentication time.

But macOS doesn't provide a ssh-askpass utility, nor can you easily modify the environment variables set with the default agent found in (protected by apple's system integrity protection):

$ cat /System/Library/LaunchAgents/com.openssh.ssh-agent.plist

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.openssh.ssh-agent</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/ssh-agent</string>
        <string>-l</string>
    </array>
    <key>Sockets</key>
    <dict>
        <key>Listeners</key>
        <dict>
            <key>SecureSocketWithKey</key>
            <string>SSH_AUTH_SOCK</string>
        </dict>
    </dict>
    <key>EnableTransactions</key>
    <true/>
</dict>
</plist>

What you could do, is create your own com.openssh.ssh-agent.plist in ~/Library/LaunchAgents that defines SSH_ASKPASS perhaps using (https://github.com/theseal/ssh-askpass) or (https://blog.nathanhigley.com/posts/hardwaresshkeysonmacos/). It would have to define a different SecureSocketWithKey environment variable like USER_SSH_AUTH_SOCK and specify that environment variable for the IdentityAgent config in your ~/.ssh/config file.

[MichaelRoosz]

I have updated the homebrew cask and formula to configure the native ssh-agent to use ssh-askpass and to set the needed env vars system-wide:

SSH_ASKPASS=/opt/homebrew/bin/ssh-askpass
SSH_SK_PROVIDER=/usr/local/lib/libsk-libfido2.dylib

so simply running

brew install michaelroosz/ssh/libsk-libfido2-install

should take care of everything now