`CertificateRequest.CreateSelfSigned` fails on macOS Sequoia

[Annny-Cap-Daniel]

Update: repro app here

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

the command dotnet dev-certs https fails with the error: There was an error creating the HTTPS developer certificate.

Exception:

An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable`1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive).

What I already tried:

• dotnet dev-certs https --clean
• installing the lastest dotnet version
• uninstalling all old dotnet versions
• reinstalling dotnet completely
• restarting the Terminal, reboot, etc.

Nothing changed the behaviour and i still get the same error.

Expected Behavior

the certificate should be created.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

8.0.401

Anything else?

dotnet --info
.NET SDK:
 Version:           8.0.401
 Commit:            811edcc344
 Workload version:  8.0.400-manifests.56cd0383
 MSBuild version:   17.11.4+37eb419ad

Runtime Environment:
 OS Name:     Mac OS X
 OS Version:  15.0
 OS Platform: Darwin
 RID:         osx-arm64
 Base Path:   /usr/local/share/dotnet/sdk/8.0.401/

.NET workloads installed:
Configured to use loose manifests when installing new manifests.
 [wasm-tools]
   Installation Source: SDK 8.0.400
   Manifest Version:    8.0.8/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.8/WorkloadManifest.json
   Install Type:        FileBased

 [wasi-experimental]
   Installation Source: SDK 8.0.400
   Manifest Version:    8.0.8/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.workload.mono.toolchain.current/8.0.8/WorkloadManifest.json
   Install Type:        FileBased

 [ios]
   Installation Source: SDK 8.0.400
   Manifest Version:    17.5.8020/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.ios/17.5.8020/WorkloadManifest.json
   Install Type:        FileBased

 [maccatalyst]
   Installation Source: SDK 8.0.400
   Manifest Version:    17.5.8020/8.0.100
   Manifest Path:       /usr/local/share/dotnet/sdk-manifests/8.0.100/microsoft.net.sdk.maccatalyst/17.5.8020/WorkloadManifest.json
   Install Type:        FileBased

Host:
  Version:      8.0.8
  Architecture: arm64
  Commit:       08338fcaa5

.NET SDKs installed:
  8.0.401 [/usr/local/share/dotnet/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 8.0.8 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

Other architectures found:
  x64   [/usr/local/share/dotnet/x64]
    registered at [/etc/dotnet/install_location_x64]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

[amcasey]

@Annny-Cap-Daniel Can you please try again with the --verbose flag? It won't fix the problem, but it should provide more information about what's going wrong. Please run dotnet dev-certs https --clean --verbose and then dotnet dev-certs https --verbose.

[Annny-Cap-Daniel]

Thanks for your prompt reply!

dotnet dev-certs https --clean --verbose outputs:

Cleaning HTTPS development certificates from the machine. This operation might require elevated privileges. If that is the case, a prompt for credentials will be displayed.
[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[6] Finished listing certificates.
[8] Filtered certificates: no certificates
[9] Excluded certificates: no certificates
HTTPS development certificates successfully removed from the machine.

and dotnet dev-certs https --verbose outputs:

[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[1] Listing certificates from CurrentUser\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[1] Listing certificates from LocalMachine\My
[2] Found certificates: no certificates
[3] Checking certificates validity
[4] Valid certificates: no certificates
[5] Invalid certificates: no certificates
[6] Finished listing certificates.
[8] Filtered certificates: no certificates
[9] Excluded certificates: no certificates
[16] No valid certificates found.
[17] Generating HTTPS development certificate.
[19] An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable`1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive).
There was an error creating the HTTPS developer certificate.

[sh0knah]

I'm getting the same thing. Same exact output.

[amcasey]

@sh0knah Are you also on Sequoia? I think we had some other cert problems when 14.4.4 came out, so it could be a system API change that we need to react to.

[amcasey]

@vcsjones Could this be another change on Apple's end?

[amcasey]

Possibly relevant: https://github.com/dotnet/aspnetcore/issues/19590#issuecomment-881652003

[sh0knah]

I am on Sequoia. Beta 7.

[amcasey]

What happens if you run this console app?

using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;

var subject = new X500DistinguishedName("CN=localhost");

var sanBuilder = new SubjectAlternativeNameBuilder();
sanBuilder.AddDnsName("localhost");

var keyUsage = new X509KeyUsageExtension(X509KeyUsageFlags.KeyEncipherment | X509KeyUsageFlags.DigitalSignature, critical: true);
var enhancedKeyUsage = new X509EnhancedKeyUsageExtension(
    [ new Oid("1.3.6.1.5.5.7.3.1", "Server Authentication") ],
    critical: true);
var basicConstraints = new X509BasicConstraintsExtension(
    certificateAuthority: false,
    hasPathLengthConstraint: false,
    pathLengthConstraint: 0,
    critical: true);

using var rsa = RSA.Create(2048);
var request = new CertificateRequest(subject, rsa, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
request.CertificateExtensions.Add(basicConstraints);
request.CertificateExtensions.Add(keyUsage);
request.CertificateExtensions.Add(enhancedKeyUsage);
request.CertificateExtensions.Add(sanBuilder.Build());

var notBefore = DateTimeOffset.UtcNow.AddDays(1);
var notAfter = notBefore.AddDays(1);
using var cert = request.CreateSelfSigned(notBefore, notAfter);
Console.WriteLine(cert is not null);

It creates a cert that's similar to the dev cert without all the extra baggage of being a dotnet tool.

[Annny-Cap-Daniel]

I'm on MacOS Sequoia (15.0 Beta 24A5327a).

Just ran the sample console app and get the same exception as in the dotnet tool:

Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain.
   at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey)
   at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey)
   at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey)
   at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter)
   at Program.<Main>$(String[] args) in /Users/user/Projects/ConsoleApp1/ConsoleApp1/Program.cs:line 28

[amcasey]

And what if you change this?

-certificateAuthority: false,
+certificateAuthority: true,

[Annny-Cap-Daniel]

same exception unfortunately.

[amcasey]

same exception unfortunately.

That's actually good news because it means they haven't decided they don't like the slightly unusual shape of our certificate.

[amcasey]

Moving this to dotnet/runtime now that there's a repro not involving aspnetcore.

[bartonjs]

I don't know when we're going to have a chance to look at this (not "indefinitely far", but probably "at least several days away"). We have a few small fires to deal with for the .NET 9 release on already in-market OSes, and are super-saturated on those tasks.

If anyone from the community at large wants to jump in and help debug what's going on, that'd be appreciated.

[oumaima-aarabe]

first try enabling verbose logging to get more detailed error messages using

export DOTNET_CLI_TELEMETRY_OPTOUT=1
export DOTNET_MULTILEVEL_LOOKUP=0
dotnet dev-certs https --trust --verbose

and several common troubleshooting steps would include to clear existing certificates manually or you use the security command to list and delete any certificates related to localhost or dotnet

security find-certificate -a
security delete-certificate -c "ASP.NET Core HTTPS development certificate"

also did you ensure that your user account has the necessary permissions to access the keychain, tried creating a certificate manually using openssl ?

[sajjadhajavi]

same problem: Mac OS Sequoia Version 15.0 Beta (24A5327a)

dotnet dev-certs https --trust --verbose

[1] Listing certificates from CurrentUser\My [2] Found certificates: no certificates [3] Checking certificates validity [4] Valid certificates: no certificates [5] Invalid certificates: no certificates [6] Finished listing certificates. Trusting the HTTPS development certificate was requested. If the certificate is not already trusted we will run the following command: 'security add-trusted-cert -p basic -p ssl -k <> <>' This command might prompt you for your password to install the certificate on the keychain. To undo these changes: 'security remove-trusted-cert <>'

[1] Listing certificates from CurrentUser\My [2] Found certificates: no certificates [3] Checking certificates validity [4] Valid certificates: no certificates [5] Invalid certificates: no certificates [6] Finished listing certificates. [1] Listing certificates from LocalMachine\My [2] Found certificates: no certificates [3] Checking certificates validity [4] Valid certificates: no certificates [5] Invalid certificates: no certificates [6] Finished listing certificates. [8] Filtered certificates: no certificates [9] Excluded certificates: no certificates [16] No valid certificates found. [17] Generating HTTPS development certificate. [19] An error has occurred generating the certificate: Interop+AppleCrypto+AppleCommonCryptoCryptographicException: The specified item is no longer valid. It may have been deleted from the keychain. at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain) at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey) at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey) at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey) at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter) at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateSelfSignedCertificate(X500DistinguishedName subject, IEnumerable1 extensions, DateTimeOffset notBefore, DateTimeOffset notAfter) at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.CreateAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter) at Microsoft.AspNetCore.Certificates.Generation.CertificateManager.EnsureAspNetCoreHttpsDevelopmentCertificate(DateTimeOffset notBefore, DateTimeOffset notAfter, String path, Boolean trust, Boolean includePrivateKey, String password, CertificateKeyExportFormat keyExportFormat, Boolean isInteractive). There was an error creating the HTTPS developer certificate.

[vcsjones]

This almost certainly looks like another lifetime issue. I should be able to take a look by the end of the week.

[vcsjones]

Before I get to it, any chance someone can try on .NET 9 preview 7 (or later)?

[vcsjones]

Okay, curiosity took over so I had a peek.

The "good news" is that our unit tests fail and reproduce the issue:

Interop+AppleCrypto+AppleCommonCryptoCryptographicException : The specified item is no longer valid. It may have been deleted from the keychain.
    Stack Trace:
     at Interop.AppleCrypto.X509CopyWithPrivateKey(SafeSecCertificateHandle certHandle, SafeSecKeyRefHandle privateKeyHandle, SafeKeychainHandle targetKeychain) in /Users/vcsjones/Projects/runtime/src/libraries/Common/src/Interop/OSX/System.Security.Cryptography.Native.Apple/Interop.X509.macOS.cs:line 279
     at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(SafeSecKeyRefHandle privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/AppleCertificatePal.Keys.macOS.cs:line 199
     at System.Security.Cryptography.X509Certificates.AppleCertificatePal.CopyWithPrivateKey(RSA privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/AppleCertificatePal.Keys.macOS.cs:line 131
     at System.Security.Cryptography.X509Certificates.RSACertificateExtensions.CopyWithPrivateKey(X509Certificate2 certificate, RSA privateKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/RSACertificateExtensions.cs:line 53
     at System.Security.Cryptography.X509Certificates.CertificateRequest.CreateSelfSigned(DateTimeOffset notBefore, DateTimeOffset notAfter) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/CertificateRequest.cs:line 519
     at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateAndTestChain(AsymmetricAlgorithm rootPrivKey, AsymmetricAlgorithm intermed1PrivKey, AsymmetricAlgorithm intermed2PrivKey, AsymmetricAlgorithm leafPubKey) in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CertificateRequestChainTests.cs:line 388
     at System.Security.Cryptography.X509Certificates.Tests.CertificateCreation.CertificateRequestChainTests.CreateChain_RSA() in /Users/vcsjones/Projects/runtime/src/libraries/System.Security.Cryptography/tests/X509Certificates/CertificateCreation/CertificateRequestChainTests.cs:line 43

The other "good news" is that this is the only thing that fails on macOS 15.

[vcsjones]

This is a behavioral change for macOS 15. I have a fix put up for it, but unfortunately there are no reasonable work around at the moment.

The core of the issue is that X509Certificate2.CopyWithPrivateKey does not work for RSA/ECDSA when the private key is an ephemeral key that was created or imported in to the internal Algorithm.Create(). On macOS, that is SecurityTransforms.

CertificateRequest uses this API itself, which is why it fails.

[ergunr]

Envoyé de mon iPhone

Le 27 août 2024 à 19:01, Larry Ewing @.***> a écrit :



Closed #106775https://github.com/dotnet/runtime/issues/106775 as completed via #106973https://github.com/dotnet/runtime/pull/106973.

— Reply to this email directly, view it on GitHubhttps://github.com/dotnet/runtime/issues/106775#event-14031453259, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AUW2LQ266QG4O27TVL2OUBTZTSPGPAVCNFSM6AAAAABM4TNM5SVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGAZTCNBVGMZDKOI. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[adityamandaleeka]

@lewing @vcsjones Just confirming, this is not getting fixed in RC1?

[vcsjones]

@adityamandaleeka the fix is not present in the release/9.0-rc1 branch, so no.

[ZephiroRB]

Hey same error MacOs Sequoia final version!

[vcsjones]

Hey same error MacOs Sequoia final version!

Yep. This will be fixed in the October patch cycle for .NET. See the corresponding announcement for additional information.

[Flight]

Any update or workaround here? Our team is blocked because of this. Should we just reinstall the old version of Mac OS just to have this command work? dotnet dev-certs https --trust

[vcsjones]

I am not aware of any work arounds. If downgrading is an option, that seems reasonable. The only other option would be to wait until the October release of .NET.

[JohnGalt1717]

This shouldn't be closed until there is a release out the door. And someone internal should be making the call to release an emergency patch release .402 that fixes this.

[ryanmendoza]

This shouldn't be closed until there is a release out the door. And someone internal should be making the call to release an emergency patch release .402 that fixes this.

Agreed. Otherwise pointing to some documented steps on how to manually generate the certification and import it using the dotnet CLI would be better than a "wait until the October update train" hand-waive.

We don't all have the luxury of not upgrading to Apple's latest OS because we're not all just dotnet developers.

[Flight]

So all the users of the modern Mac OS is blocked from using .net and you just tell to wait for the October release? Guys, it looks like a critical issue and the fix should be deployed in hours, not even days...

[Winter979]

Dont think its the best but it works for the time being. Generating a self signed crt + key, convert it to a pfx and then configure the app to use that

appsettings.Development.json

"Kestrel": {
  "Endpoints": {
    "Https": {
      "Url": "https://localhost:7209",
      "Certificate": {
        "Path": "../Certs/localhost.pfx",
      }
    }
  }
}

At least i can now continue development.

[octo888]

Dont think its the best but it works for the time being. Generating a self signed crt + key, convert it to a pfx and then configure the app to use that

appsettings.Development.json

"Kestrel": {
  "Endpoints": {
    "Https": {
      "Url": "https://localhost:7209",
      "Certificate": {
        "Path": "../Certs/localhost.pfx",
      }
    }
  }
}

At least i can now continue development.

it helps me to run webapp (but I had to add config in the Program.cs) but I still can't make requests from android emulator. I am shocked that this issue was closed and that's it, wait until November

[pvasek]

Workaround that worked for me:

1. download the tar.gz version of nightly build of upcoming dotnet, can be found in package-table.md
2. Unpack it
3. Go to that unpacked folder
4. Run ./dotnet dev-certs https --trust (it's important to use ./ otherwise it use the installed dotnet)

After that it ask for password install the certificate and debugging of my 8.0 app works again.

[kalebzettl]

@pvasek Thank you mate, that worked for me too, only thing I had to quarantine the folder, I have SIP enabled and all macOS wanted to do was complain about it being untrusted 🙄

xattr -d com.apple.quarantine -r dotnet-sdk-9.0.100-rc.2.24473.20-osx-arm64 cd dotnet-sdk-9.0.100-rc.2.24473.20-osx-arm64 ./dotnet dev-certs https --trust

[MichaelCharles]

@kalebzettl @pvasek Thanks, that worked for me. Here's a consolidated instruction set:

1. Just in case, delete any certs that currently exist. Open a terminal and run: dotnet dev-certs https --clean

2. Download the tar.gz file of the "main" release from the .NET SDK package table. You can also access the links directly below.

   • Apple Silicon.
   • Intel

3. Unpack the downloaded file.

4. Remove the quarantine attribute from the unpacked folder. From your terminal run: xattr -d com.apple.quarantine -r <folderName> Replace <folderName> with the name of your unpacked folder. For example: xattr -d com.apple.quarantine -r dotnet-sdk-9.0.100-rc.2.24473.22-osx-arm64

5. Navigate to the unpacked folder: cd dotnet-sdk-9.0.100-rc.2.24473.22-osx-arm64

6. From within this folder, run the following to generate and trust the certificate. ./dotnet dev-certs https --trust