dotnet-policy-service[bot]
commented
1 month ago Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones See info in area-owners.md if you want to be subscribed.
Closed dtivel closed 3 days ago
dotnet-policy-service[bot]
commented
1 month ago Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones See info in area-owners.md if you want to be subscribed.
bartonjs
commented
1 month ago https://github.com/dotnet/runtime/blob/676189ad5e60ce6ed3337120f29470bff38bad8d/src/native/libs/System.Security.Cryptography.Native.Apple/pal_signverify.c#L154C9-L222 doesn't have any loops in it; the most likely place for it to be hanging is SecKeyCreateSignature.
It seems like Apple's Crypto stack is gummed up. I assume you've rebooted since the failure?
I'm not sure there's anything we can do here... but maybe @vcsjones or @filipnavara have suggestions/insights?
dtivel
commented
1 month ago @bartonjs, they're CI machines in a macOS machine pool. I don't have reboot capabilities. Furthermore, per AzDO logs, the failures are not isolated to one machine; it's multiple.
filipnavara
commented
1 month ago We have often seen similar issue when there was an interactive UI asking to unlock the keychain (which is obviously not visible on CI and requires remote access to the machine). The access to the keychain can be unlocked using the security tool from command line / CI script. We also observed that after macOS updates the password for the default keychain gets reset if you change it manually to something like empty password. Like OP, this appeared to be inconsistent between our macOS x64 and macOS arm64 (self-hosted) CI machines.
vcsjones
commented
1 month ago A blocking keychain modal does seem like the most plausible culprit here.
I simplified the repro with a copy of
When you repro'ed, were you SSHed or Remote Screen Sharing in to the Mac, or were you actually logged-in in front of the computer?
dtivel
commented
1 month ago The access to the keychain can be unlocked using the security tool from command line / CI script.
@filipnavara, with the security unlock-keychain command, what file path would I provide? Note my repro where I'm using CertificateAuthority to create RSA keys on the fly. I'm not trying to access anything persistent.
When you repro'ed, were you SSHed or Remote Screen Sharing into the Mac, or were you actually logged-in in front of the computer?
@vcsjones, this is entirely within a CI pipeline context. I'm not remoted into any machine. I'm just queuing builds, which run tests and hit this problem.
filipnavara
commented
1 month ago There’s no path, the default is the login keychain. This is literally the code in our script: security unlock-keychain -p "${{ secrets.AGENT_KEYCHAIN_PASSWORD }}". Note that macOS keychain APIs (and the .NET X509 APIs by proxy) often use the keychain even if you don’t store the actual keys. It may create temporary keychains which should be unique and unlocked but YMMV.
jeffhandley
commented
1 month ago @dtivel This issue isn't currently actionable for us. Please let us know if you identify a cause other than a blocking keychain modal.
dotnet-policy-service[bot]
commented
1 month ago This issue has been marked needs-author-action and may be missing some important information.
dotnet-policy-service[bot]
commented
2 weeks ago This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.
dotnet-policy-service[bot]
commented
3 days ago This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.
Description
I have some test code that recently started hanging in macOS x64 CI builds.
With
dotnet test --blame-hang-timeout 60sI captured a hang dump with the following callstack:Reproduction Steps
My original repro uses a fork of /src/libraries/Common/tests/System/Security/Cryptography/X509Certificates/CertificateAuthority.cs and is complicated.
I simplified the repro with a copy of /src/libraries/Common/tests/System/Security/Cryptography/X509Certificates/CertificateAuthority.cs and the following unit test:
Note that this only hangs on macOS x64. The same repro on macOS amd64 works fine. It also works fine on Windows and Linux.
Expected behavior
No hang.
Actual behavior
Hang almost every time.
Regression?
Yes, but I cannot pinpoint a specific change. This succeeded on June 18th, 2024 with .NET SDK 8.0.302, still on macOS 12.7.5 x64. Subsequent runs started failing.
I don't want to imply that the regression was from moving from 8.0.302 to 8.0.303, since the first failure was on 8.0.302.
Known Workarounds
No response
Configuration
macOS 12.7.5 x64 .NET SDK 8.0.303
Other information
I have a MacBook Air also running macOS 12.7.5 x64 and cannot reproduce the hang locally.