maxgoedjen
commented
2 years ago Yikes, that's not good - first time I'm seeing this one. Guessing you've got a bunch of other macOS issues going on right now too? Assuming you've already tried rebooting?
Open sminnee opened 2 years ago
maxgoedjen
commented
2 years ago Yikes, that's not good - first time I'm seeing this one. Guessing you've got a bunch of other macOS issues going on right now too? Assuming you've already tried rebooting?
maxgoedjen
commented
2 years ago Also: any changes recently in security settings? Adding/removing fingers to Touch ID, changing password etc?
maxgoedjen
commented
2 years ago @sminnee 👋 hey just checking in on this one, still happening?
andrea-sdl
commented
2 years ago @maxgoedjen same error is happened to me today. Rebooting fixed it but it's not very good when it happens :D
FYI I didn't change anything related to touchid/password. I remember allowing a new app into the accessibility settings but I've done it frequently in the past without this error happening.
sminnee
commented
2 years ago Heya I got this working again after some combination of reboots and OS upgrades.
maxgoedjen
commented
2 years ago Glad to hear that. When you peeps were seeing that state, was it just empty? Or was it telling your Mac had no Secure Enclave?
maxgoedjen
commented
2 years ago Anyone who sees in future: would love a screenshot 🙏
andrea-sdl
commented
2 years ago Glad to hear that. When you peeps were seeing that state, was it just empty? Or was it telling your Mac had no Secure Enclave?
It was an empty list. I opened secretive and there were no elements in it.
Will post a screenshot if it happens again. Is there anything else I might do to give you more information?
andrea-sdl
commented
2 years ago Here it is ;) Using Secretive Version 2.2.0 (1.1857237470)
I only have one secret in secretive and it doesn't appear here
maxgoedjen
commented
2 years ago @andrea-sdl thanks for the screenshot, that's very helpful (and very weird). I was expecting the Mac to just report that it didn't have a SEP to be perfectly honest.
sdemjanenko
commented
9 months ago I've been running into this issue off and on since installing the Rippling MDM agent and changing my Mac password. Today it came to a head. I think I found the resolution which is the "Local items" keychain was locked (presumably with my old password) and I wasn't able to put it in an unlocked state. For some reason, in the leadup to this, an adobe CC background process would prompt me for access to that chain and somehow that would unlock it, letting Secretive work.
I think it came to a head today because something in adobe updated and I was no longer getting those prompts to unlock that keychain and I had no way to get the prompts to show, and therefore I was locked out.
I had to delete the LocalItems keychain and restart. That then seems to have fixed the issue (i'll confirm in a few days if it stays fixed). On the downside, I lost what was in there.
This is a screenshot after fixing. Previously, the "login" chain was unlocked, but the "Local Items" chain was locked which is weird and suggests the "Local Items" chain isn't using the login password to the mac.
fsouza
commented
6 months ago @maxgoedjen hi there, thanks for building Secretive. I started running into this issue today. Rebooting seems to help, but let me know if there's any information you'd like me to collect the next time it happens :)
fsouza
commented
6 months ago Ok, upon further investigation, it seems that the issue is triggered by trying to use the key with the laptop locked?
Here's some information about the system:
％ uname -a
Darwin ... 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:54:51 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6030 arm64 arm DarwinReproducing steps:
％ ssh -l git -T github.com
Hi fsouza! You've successfully authenticated, but GitHub does not provide shell access.％ sleep 30 && ssh -l git -T github.com％ sleep 30 && ssh -l git -T github.com
sign_and_send_pubkey: signing failed for ECDSA "ecdsa-sha2-nistp256" from agent: agent refused operation
git@github.com: Permission denied (publickey).
Again, this failure is expected as the Secure Enclave is not available while the computer is locked, but now that the computer is unlocked, ssh -T still doesn't work:
％ ssh -l git -T github.com
sign_and_send_pubkey: signing failed for ECDSA "ecdsa-sha2-nistp256" from agent: agent refused operation
git@github.com: Permission denied (publickey).Checking Secretive, no secrets are listed:
If I restart the agent, secrets are still not available. They only come back after I restart the computer. If I try to add a new secret, Secretive crashes just like reported by the OP. Crash report: https://gist.github.com/fsouza/675c9ddc54f8e57393f9890bb269280b
I can reliably reproduce the issue with the steps listed above. I haven't found a way to recover from this state besides rebooting, so I haven't done this more than a couple of times heh
I realize that the Secure Enclave is not available while the computer is locked, but is there anything that can be done to prevent Secretive from getting stuck in the bad state? Or a more effective way to reset it that doesn't require rebooting the machine? Maybe some way to force unlock the Keychain used by Secretive? (via a special user signal maybe? Or some initialization check that would allow it to unlock when I restart the agent)
maxgoedjen
commented
6 months ago That is FANTASTIC @fsouza, I'll give that a go in a bit, I've had absolutely no luck reproducing this consistently, really appreciate the detailed steps, thanks!
fsouza
commented
6 months ago Just one more thing and then I'll go to bed: logging out and logging in again seems to fix it too, no need to reboot.
maxgoedjen
commented
6 months ago Hm, having trouble reproducing it here. I can repro the "locked machine failed the request" bit but on unlock and retry it works fine.
% sleep 30 && ssh -l git -T github.com
sign_and_send_pubkey: signing failed for ECDSA "ecdsa-sha2-nistp256" from agent: agent refused operation
git@github.com: Permission denied (publickey).
% ssh -l git -T github.com
Hi maxgoedjen! You've successfully authenticated, but GitHub does not provide shell access.I see from your crash report you're on the same OS as me (14.2, at time of writing). I've tried locking while connected to external screen, while in standalone laptop, and closing lid. Any other detail I could be missing?
fsouza
commented
6 months ago Yeah I've noticed the issue happens as soon as I lock my computer, regardless of whether or not I run ssh with the laptop locked. This didn't use to happen 😞 (note I've been using Secretive for one week)
I've been using the laptop with the lid closed, connected to an external monitor, keyboard and mouse, the things that changed since I first setup Secretive are:
So I started suspecting touch ID and that seems to be the issue?
So it seems that with the laptop closed and no touchid-enabled device, the Enclave never unlocks once it's been locked? (probably because touch ID is effectively disabled?). Is this expected? Like, is Touch ID a requirement and I just missed that? 🙈 If that's the case, I apologize for wasting your time 😭
On a side note, as mentioned by @sdemjanenko, the "Local Items" keychain behavior matches the behavior of Secretive (i.e. it's unlocked in cases (1) and (2), but remains locked in (3)).
maxgoedjen
commented
6 months ago my location (and timezone). I assume this is not relevant as other folks in my organization have gone through the same process and don't get into this state. Including it for completeness. the monitor it connects to. I don't think this can be an issue, but I'm listing it for completeness. the keyboard it connects to (it used to connect with an apple external keyboard with touch id, but not anymore, and therefore I'm no longer using touch id)
The keyboard might be relevant there? But probably mostly to the extent of "unlocking it with password, not Touch ID." I'll try that, I think I've been unlocking with TID every time so far.
So it seems that with the laptop closed and no touchid-enabled device, the Enclave never unlocks once it's been locked? (probably because touch ID is effectively disabled?). Is this expected? Like, is Touch ID a requirement and I just missed that? 🙈 If that's the case, I apologize for wasting your time 😭
That definitely shouldn't be the case. You can definitely use it on non-TID Macs (I personally used it on a Mac mini with non-TID keyboard before the M1 MBPs came out)
fsouza
commented
6 months ago I have no idea of what's going on anymore: after the recent upgrade (Sonoma 14.2.1), I can no longer reproduce the issue 🤔
fsouza
commented
5 months ago Even though I can no longer reproduce the issue, other folks in the org are running into the same issue (secretive works when unlocked with touch id, but doesn't when unlocked with the password), even though they're also on 14.2.1 😭 Any additional ideas on how we can debug this?
maxgoedjen
commented
5 months ago @fsouza I'd be curious if they see anything unusual/mentioning the SEP in Console.app (Secretive-related or otherwise). If there's anyone encountering this that'd be brave enough to fire up Xcode and gather some debug info, I'd definitely be happy to figure out some stuff to try there.
fsouza
commented
5 months ago If there's anyone encountering this that'd be brave enough to fire up Xcode and gather some debug info, I'd definitely be happy to figure out some stuff to try there.
I'll see if I can find some volunteers. Too bad I can no longer repro the issue. While I look for volunteers, any pointers on what debugging information would be useful and how we can gather it? Is the idea to run Secretive from some branch?
maxgoedjen
commented
5 months ago Mainly I'd want to be stepping through https://github.com/maxgoedjen/secretive/blob/main/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L228 and https://github.com/maxgoedjen/secretive/blob/main/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L36 and also keeping an eye on the console to see if there's anything interesting.
grugnog
commented
5 months ago Using Kandji MDM here and had this issue after I needed to do a password verification after resume. I did not have to do this after a normal resume where I unlock with Touch ID. After a restart the key was present again.
fsouza
commented
4 months ago Mainly I'd want to be stepping through https://github.com/maxgoedjen/secretive/blob/main/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L228 and https://github.com/maxgoedjen/secretive/blob/main/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L36 and also keeping an eye on the console to see if there's anything interesting.
I got lucky and can now reproduce the issue again lol I'm not sure what changed, my laptop "soft-crashed" (keyboard and mouse stopped responding, screen went dark, but sound was still on), then I manually restarted it and when it came back, I could repro the issue again: if I unlock with the password, secrets are not available, if I use Touch ID, everything is fine.
I stepped through creation and retrieval of secrets. I'll poke around the code a bit with help from AI™, but I'm posting an update here just in case the error is obvious to anyone reading this message.
First, creating a new secret fails with the following error:
Secretive/CreateSecretView.swift:48: Fatal error: 'try!' expression unexpectedly raised an error: Error Domain=NSOSStatusErrorDomain Code=-25308 "failed to generate asymmetric keypair" (errKCInteractionNotAllowed / errSecInteractionNotAllowed: / Interaction is not allowed with the Security Server.) UserInfo={numberOfErrorsDeep=0, NSDescription=failed to generate asymmetric keypair}That error is thrown here: https://github.com/maxgoedjen/secretive/blob/c7983bbf33d2111d85eac8ae6b5cdbb643ef97bb/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L65-L67
loadSecrets doesn't throw any errors, it returns from this guard: https://github.com/maxgoedjen/secretive/blob/c7983bbf33d2111d85eac8ae6b5cdbb643ef97bb/Sources/Packages/Sources/SecureEnclaveSecretKit/SecureEnclaveStore.swift#L239
Because publicUntyped is nil.
maxgoedjen
commented
4 months ago Hm... that lines up with the other reports but doesn't give too much more additional information – basically the system just seems to be in a bad state, I'm not sure if there's anything we can do specifically to shake it loose. Anything interesting in Console.app (or Xcode console) about keychain?
fsouza
commented
4 months ago I don't know why I didn't think of this before, but security unlock-keychain gets things rolling again, which is I guess yet another indication of what we already suspected here in this thread.
The only things I see in the Xcode console are:
This method should not be called on the main thread as it may lead to UI unresponsiveness.
This method should not be called on the main thread as it may lead to UI unresponsiveness.
This method should not be called on the main thread as it may lead to UI unresponsiveness.
This method should not be called on the main thread as it may lead to UI unresponsiveness.
This method should not be called on the main thread as it may lead to UI unresponsiveness.
This method should not be called on the main thread as it may lead to UI unresponsiveness.
FAULT: <NSRemoteView: 0x152f51df0 com.apple.TextInputUI.xpc.CursorUIViewService TUICursorUIViewService> determined it was necessary to configure <TUINSWindow: 0x132e49d60> to support remote view vibrancyConsole.app shows nothing interesting, either when I lock/unlock the laptop, or open/close Secretive.
maxgoedjen
commented
4 months ago Hadn't occurred to me either, but interesting thing to try.
Think the "best" solution here might be specifically watching for the interaction not allowed message and showing a "restart your Mac" error so people don't freak out and think there's data loss. Less than ideal but I'm kinda skeptical we'll be able to do much beyond that.
fsouza
commented
4 months ago Do you think it's feasible to have Secretive detect that the Keychain/Secure Enclave is locked and try to unlock it automatically (which would require user authentication)? I haven't looked into the details of how it would work yet, but the idea is that we'd only want to do that if the screen is not locked, and hopefully we'd want to try to do it when the user tries to use the secrets.
So, for example, when I run ssh -T git@github.com, rather than failing, Secretive would detect that the Secure Enclave is locked and try to unlock it (we'd need to differentiate between having no secrets and not having access to the Keychain stuff).
If you think that's an acceptable feature, I can try to flush it out a bit more to confirm what's feasible and potentially send a PR.
fsouza
commented
4 months ago (fun fact: apparently, security lock-keychain on a random keychain that is locked helps too, it looks like any keychain operation fixes the issue, which probably points to some weird state getting out of sync within macOS 😭)
maxgoedjen
commented
4 months ago I think I feel differently about different parts of this:
Overall definitely feel free to do some POC work here – I'd be interested to see what's doable.
fsouza
commented
3 months ago A quick update on this, we continue to investigate it and are trying to get more information to escalate. We inspected the logs when unlocking computers to see if we'd find anything useful, and we did. Will keep digging.
On affected laptops, we see:
loginwindow: [com.apple.loginwindow.logging:Standard] -[LWScreenLockAuthentication _authSuccessUsingPassword:forUser:] | Screen saver unlocked by admin, did NOT unlock the user's keychainOn non-affected laptops, we see:
loginwindow: [com.apple.loginwindow.logging:Standard] -[LWScreenLockAuthentication _authSuccessUsingPassword:forUser:] | Unlock succeeded, with password, attempting to unlock the login keychain
...
loginwindow: [com.apple.loginwindow.logging:Standard] -[LWKeychainSupport unlockLoginKeychainFromScreenLock:pwd:] | SecKeychainLogin returned success
loginwindow: [com.apple.loginwindow.logging:Standard] -[LWKeychainSupport unlockLoginKeychainFromScreenLock:pwd:] | returning; 1
loginwindow: [com.apple.loginwindow.logging:Standard] -[LWScreenLockAuthentication _authSuccessUsingPassword:forUser:] | Unlock succeeded, with password, attempting to unlock the keybag
...(and that succeeds)
Sharing in case other folks have ideas, but we'll continue to explore investigate on our side here :)
(note: we check the logs with something like log show --predicate "process == 'loginwindow'" --last 1m after unlocking, and search for _authSuccessUsingPassword:forUser:)
I started Secretive today, and:
It seems like both issues are caused by the same issue: Secretive can't access the Security Server.
OS: macOS Monterey 12.1 Hardware: M1 Air
Crash report below, I don't believe I've left any PII in this...