crypto/x509: "certificate is not standards compliant" on MacOS

dims commented 2 years ago

We hit an error with a unit test we had in Kubernetes and started looking at the impact on end users of kubernetes if the problem is not resolved by the time kubernetes 1.24 is released. More context: please see Kubernetes issue - https://github.com/kubernetes/kubernetes/issues/108956

What version of Go are you using (`go version`)?

$ go version
go version go1.18 darwin/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/dims/Library/Caches/go-build"
GOENV="/Users/dims/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/dims/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/dims/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.18/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.18/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.18"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/qw/pkzvlrfs7rn7h6r1x7r57_rw0000gn/T/go-build1513460199=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Please see https://go.dev/play/p/w4rr43vQv7d

What did you expect to see?

success

What did you see instead?

error: x509: “no-sct.badssl.com” certificate is not standards compliant

dims commented 2 years ago

The error seems be introduced here: https://github.com/golang/go/commit/feb024f4153395e5bbb2a51bb3d1ddc4f5b0d2dc#diff-9e2a37df9605e8b207365b51999e6b14e1f5db72b27ad33514dbac502d477c25R212

@liggitt summarized the ask here : https://github.com/kubernetes/kubernetes/issues/108956#issuecomment-1081011567 and https://github.com/kubernetes/kubernetes/issues/108956#issuecomment-1080972473 on what impact this will have on kubernetes users for whom there was no issues before we switched to go1.17 when they try the kubernetes kubectl command built using go1.18.

Worst case we would like to document scenarios under which users will hit the certificate is not standards compliant error that they were not hitting before.

thanks!

seankhliao commented 2 years ago

cc @golang/security

FiloSottile commented 2 years ago

Can you share the actual affected certificate? You are unlikely to be hitting the same root cause as no-sct.badssl.com in a unit test, because SCT rules are only enforced for WebPKI certificates.

liggitt commented 2 years ago

We only noticed the issue in a unit test checking a cert we expected to be considered invalid (it was a negative test of a cert not signed by a trusted root), so at first we thought we just needed to make the unit test more tolerant of various error messages.

In spelunking around the change in the message, we also ran across https://groups.google.com/g/golang-nuts/c/RGghq2gTWss/m/7GsudTfCAgAJ which indicated requests that previously succeeded were now failing.

Before papering over the change in our unit test by tolerating more validation error messages, I wanted to understand more about which certificates the go validator considers valid that the macOS validator does not. (https://github.com/kubernetes/kubernetes/issues/108956#issuecomment-1080972473)

rolandshoemaker commented 2 years ago

Apple enforces their SCT requirements on all publicly trusted certificates as part of its base TLS policy (which we use via SecPolicyCreateSSL, since we are generally targeting the web PKI.) Publicly trusted certificates that lack embedded SCTs are very rare, making up something like 0.01% of all publicly trusted certs, but they are out there (the AWS example being probably the most common.)

This will only affect users who are using the bare system certificate pool and are validating certificates which totally lack embedded SCT, with the server providing them via a TLS extension.

It is noted in a TODO in crypto/x509/root_darwin.go that we may want to support passing SCTs passed this way, since we have no way of telling Apple to disable this particular policy (I would need to double check, but it's possible these requirements are not enforced if you use SecPolicyCreateBasicX509, but that would likely also disabled all of the other web PKI policies that we want applied), but how to do that is rather nuanced (since we'd only do anything with them on macOS.)

rolandshoemaker commented 2 years ago

Slight side note: the AWS case is a weird one, because I don't think they are sending SCTs at all, despite using publicly trusted certs, so even implementations that know how to pipe SCTs passed via TLS extensions wouldn't work on macOS 🤷.

FiloSottile commented 2 years ago

I don't think the SCT policy explains the new error in https://github.com/kubernetes/kubernetes/issues/108956 though, because that's not a publicly trusted certificate. If you want to extract that certificate and share it with us, we can tell you why that one is failing, too.

https://github.com/kubernetes/kubernetes/issues/108956#issuecomment-1080972473Before papering over the change in our unit test by tolerating more validation error messages, I wanted to understand more about which certificates the go validator considers valid that the macOS validator does not.

We can't really answer this exhaustively, because the macOS verifier has a number of evolving policies that change between OS versions. Note that the platform verifier is only used when the system roots are involved, so behaving like the system is what's expected. I assume k8s clusters use private CAs configured through config.RootCAs for most purposes, which would be unaffected by this.

FiloSottile commented 2 years ago

Slight side note: the AWS case is a weird one, because I don't think they are sending SCTs at all, despite using publicly trusted certs, so even implementations that know how to pipe SCTs passed via TLS extensions wouldn't work on macOS 🤷.

Customers should probably reach out to AWS about this. As a short term workaround, it should be possible to add the Amazon root CAs to a x509.SystemCertPool() and use it as config.RootCAs so that the Go verifier is used as well as the system one. (Don't start with an empty pool so that if the root changes you have a chance at not breaking.)

liggitt commented 2 years ago

It looks like this change means we no longer get typed TLS errors (e.g. x509.UnknownAuthorityError) when validating using system roots.

That means that special handling of those errors (logging or other fallback paths) that previously worked no longer works in go1.18.

edit: I'll open a separate issue for that, since that is distinct from the "certificate considered valid in go1.17" → "certificate considered invalid in go1.18" issue

liggitt commented 2 years ago

opened https://github.com/golang/go/issues/52010 for the untyped error issue

liggitt commented 2 years ago

I assume k8s clusters use private CAs configured through config.RootCAs for most purposes, which would be unaffected by this.

I also expect that to be true in most scenarios (and in scenarios where it isn't for the certs issued by public CAs to be compatible with system roots, though the referenced AWS issue is evidence that not all certs issued by public CAs are valid).

For k8s' use, I don't think this issue is very significant.

rolandshoemaker commented 2 years ago

Oh, that particular test certificate (in TestTLSConfig) is non-compliant in a handful of ways. It's self-signed, but isCA is false, it is missing the cert sign key usage, and it's validity period is likely too long (although I'm not sure if macOS enforces that for self-signed certs.)

calvinbui commented 2 years ago

We've had the same issue with connecting to AWS Elasticache Redis servers. Amazon will not support SCTs to avoid publishing customer cluster names in a public log. The connection previously worked fine in 1.17.

jimidle commented 2 years ago

A little more background about AWS, or at least how we were connecting to the Neptune graph database service. In case it helps anyone.

Because Neptune is a little "light" on security, you can only connect to it through local/private VPC. This isn't very useful for developers, so we have a VPN to a bastion host for a development only instance of Neptune (Neptune does not have any local installation - it is an AWS service only).

It seems that AWS did not feel the need to put any SCTs in to the Neptune cert, thinking it would only see connections from the secured VPC, and so our connections (via go, it is fine from Java for instance) will fail.

We have raised a ticket with AWS about this. There isn't much can be done about that in Go.

As this is a developer only connection, we have created a reverse proxy with a local CA root. This allows the connection for developers. Hokey, but does what we want for a developer connection. The real solution is of course for AWS to re-issue their certificate, however they say they don't want SCTs in order to avoid placing customer cluster names in a public log (see https://github.com/golang/go/issues/51991#issuecomment-1083757044 )

bcmills commented 2 years ago

This is showing up on the darwin-amd64-10_15 builder as well, though curiously not on any of the other darwin builders.

Marking as release-blocker for Go 1.19 because darwin/amd64 is a first class port.

greplogs --dashboard -md -l -e 'certificate is not standards compliant'

rolandshoemaker commented 2 years ago

As far as I can tell this seems, possibly, (this is unbearably painful to diagnose) to be an issue with 10.15.1, which is what the the darwin-amd64-10_15 builder is running. I suspect that updating the builder to use 10.15.6 would fix this, but I have absolutely no clue how viable that is.

bcmills commented 2 years ago

I suspect that updating the builder to use 10.15.6 would fix this, but I have absolutely no clue how viable that is.

@golang/release, can you weigh in on that? (How hard is the macOS 10.15 image to update?)

heschi commented 2 years ago

For amd64 I think it's maybe a day's work, if we're willing to cut over all the builders at once. Rolling it out gradually will be more unpleasant. I haven't read this issue to judge whether it's a good use of time.

rolandshoemaker commented 2 years ago

I don't think there is really any other way to address this issue, given how deeply integrated the TLS client is in the toolchain there isn't really any (safe) way of silently handling/skipping these failures. It's not a high frequency flake though (it seems somewhat correlated with when new certificates are issued) so probably not super high priority.

FiloSottile commented 2 years ago

(it seems somewhat correlated with when new certificates are issued)

Can the machine reach the internet? That sounds consistent with a bloom filter window miss on the Apple Valid system, which leads to an OCSP connection to the CA. If that fails, I could see it leading to a vague error like this.

jbg commented 2 years ago

In some cases certificates may be deliberately excluded from CT logs to avoid publishing a detailed map of internal infrastructure. (In our case, the certs are associated with DNS names that are only resolvable internally, and which resolve to private IPs.)

e.g. AWS ACM allows disabling CT logging for this purpose, which will result in a valid certificate issued by a trusted CA but not listed in CT logs.

When trying to access a service with such a cert from Go (in our case, using a Terraform provider) on developer (darwin_arm64) machines, we get this certificate is not standards compliant error.

Is there any solution other than logging the certs? Is there any knob in the Go TLS client for turning off the check, which the TF provider could provide a config option to turn?

jimidle commented 2 years ago

In our case, we were just needing developers to access a development version of Neptune on AWS. We were doing that with a bastion host and ssh. The Neptune cert is set up exactly as you say, to avoid public logs for what is intended to be a system accessed from a VPC in AWS. We therefore wrote a small reverse proxy instead of a simple tunnel. That works for our case as our code runs in the VPC as AWS intend, other than for developers.

As I understand it from the comments here, Go 1.19 will try to address this, but I cannot speak for the Go development team of course.

On May 11, 2022 at 12:39:06 PM, Jasper @.***> wrote:

In some cases certificates may be deliberately excluded from CT logs to avoid publishing a detailed map of internal infrastructure. e.g. AWS ACM allows disabling CT logging for this purpose, which will result in a valid certificate issued by a trusted CA but not listed in CT logs.

When trying to access a service with such a cert from Go (in our case, using a Terraform provider) on developer (darwin_arm64) machines, we get this certificate is not standards compliant error.

Is there any solution other than logging the certs? Is there any knob in the Go TLS client for turning off the check, which the TF provider could provide a config option to turn?

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/51991#issuecomment-1123178755, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7TMBSJBB4IPNDNOYUWPLVJM2WVANCNFSM5R4DNYCQ . You are receiving this because you commented.Message ID: @.***>

neild commented 2 years ago

Split off the builder flakes into a separate issue: #52854

bradfitz commented 2 years ago

As another example, I can't seem to hit dns.nextdns.io from macOS + Go 1.18 anymore:

bradfitz@book1pro nextdns % go version
go version go1.18.3 darwin/arm64

bradfitz@book1pro nextdns % sw_vers 
ProductName:    macOS
ProductVersion: 12.4
BuildVersion:   21F79

bradfitz@book1pro nextdns % cat nextdns.go
package main

import (
    "log"
    "net/http"
    "os"
)

func main() {
    res, err := http.Get("https://dns.nextdns.io/")
    if err != nil {
        log.Fatal(err)
    }
    res.Write(os.Stdout)
}

bradfitz@book1pro nextdns % go run nextdns.go
2022/07/10 19:14:53 Get "https://dns.nextdns.io/": x509: “dns.nextdns.io” certificate is not standards compliant
exit status 1

/cc @rs

rs commented 2 years ago

Recently, ZeroSSL stopped embedding SCTs in their certs if must staple was requested in the CSR. On macOS and iOS it generates this error when used with the system provided SSL implementation; Safari is fine though. My theory is that Safari is doing the extra work of fetching the SCTs when they are not embedded, but we couldn't find the right code to do that in custom client code.

We are in contact with Sectigo — ZeroSSL's upstream CA — trying to convince them to revert to embedded SCTs, but we have little hope. We may end up switching CA altogether.

Go is affected in certain conditions when crypto/x509 code use the macOS/iOS verifier implementation instead of Go's probably due to this. I'm not sure why, but when Go is using a custom root cert pool, this problem does not seem to happen:

package main

import (
    "crypto/tls"
    "crypto/x509"
    "log"
    "net/http"
    "os"
)

func main() {
    rootCAs, _ := x509.SystemCertPool()
    rootCAs.AppendCertsFromPEM(zerossl)
    cl := http.Client{
        Transport: &http.Transport{
            TLSClientConfig: &tls.Config{
                RootCAs: rootCAs,
            },
        },
    }
    res, err := cl.Get("https://dns.nextdns.io/")
    if err != nil {
        log.Fatal(err)
    }
    res.Write(os.Stdout)
}

var zerossl = []byte(`-----BEGIN CERTIFICATE-----
MIIDhTCCAwygAwIBAgIQI7dt48G7KxpRlh4I6rdk6DAKBggqhkjOPQQDAzCBiDEL
MAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNl
eSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMT
JVVTRVJUcnVzdCBFQ0MgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMjAwMTMw
MDAwMDAwWhcNMzAwMTI5MjM1OTU5WjBLMQswCQYDVQQGEwJBVDEQMA4GA1UEChMH
WmVyb1NTTDEqMCgGA1UEAxMhWmVyb1NTTCBFQ0MgRG9tYWluIFNlY3VyZSBTaXRl
IENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAENkFhFytTJe2qypTk1tpIV+9QuoRk
gte7BRvWHwYk9qUznYzn8QtVaGOCMBBfjWXsqqivl8q1hs4wAYl03uNOXgFu7iZ7
zFP6I6T3RB0+TR5fZqathfby47yOCZiAJI4go4IBdTCCAXEwHwYDVR0jBBgwFoAU
OuEJhtTPGcKWdnRJdtzgNcZjY5owHQYDVR0OBBYEFA9r5kvOOUeu9n6QHnnwMJGS
yF+jMA4GA1UdDwEB/wQEAwIBhjASBgNVHRMBAf8ECDAGAQH/AgEAMB0GA1UdJQQW
MBQGCCsGAQUFBwMBBggrBgEFBQcDAjAiBgNVHSAEGzAZMA0GCysGAQQBsjEBAgJO
MAgGBmeBDAECATBQBgNVHR8ESTBHMEWgQ6BBhj9odHRwOi8vY3JsLnVzZXJ0cnVz
dC5jb20vVVNFUlRydXN0RUNDQ2VydGlmaWNhdGlvbkF1dGhvcml0eS5jcmwwdgYI
KwYBBQUHAQEEajBoMD8GCCsGAQUFBzAChjNodHRwOi8vY3J0LnVzZXJ0cnVzdC5j
b20vVVNFUlRydXN0RUNDQWRkVHJ1c3RDQS5jcnQwJQYIKwYBBQUHMAGGGWh0dHA6
Ly9vY3NwLnVzZXJ0cnVzdC5jb20wCgYIKoZIzj0EAwMDZwAwZAIwJHBUDwHJQN3I
VNltVMrICMqYQ3TYP/TXqV9t8mG5cAomG2MwqIsxnL937Gewf6WIAjAlrauksO6N
UuDdDXyd330druJcZJx0+H5j5cFOYBaGsKdeGW7sCMaR2PsDFKGllas=
-----END CERTIFICATE——`)

go run nextdns.go
HTTP/1.1 400 Bad Request
Content-Length: 1
Content-Type: text/plain; charset=utf-8
Date: Mon, 11 Jul 2022 09:24:09 GMT
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Content-Type-Options: nosniff

AGWA commented 2 years ago

The unusual (but 100% compliant) thing about dns.nextdns.io is that SCTs are being delivered via an extension in the stapled OCSP response rather than embedded in the certificate. Safari will pass the SCTs from the stapled OCSP response to the verifier, but as the TODO indicates, Go does not do this.

This is a Go bug, not anything wrong with dns.nextdns.io or Sectigo. Sectigo omits embedded SCTs when Must-Staple is set because Must-Staple is a guarantee that the certificate will be used with OCSP stapling, so SCTs can be delivered that way instead.

rs commented 2 years ago

@AGWA I agree, there is nothing wrong in our cert and it is a Go bug. However, from our experience, this is not only breaking Go but many other TLS clients based on Apple's certificate verification code (including our own Swift app). This will require us to disable "must staple" in our certs so Setigo stop putting SCTs in the OCSP part, as many clients in the wild (some not upgradable) are broken as a consequence.

I believe Setigo should generate less unusual certificates for better compatibility.

rs commented 2 years ago

FYI we removed "must staple" on dns.nextdns.io in order to have ZeroSSL embedding SCTs. We created go51991.dns.nextdns.io with a ZeroSSL+must staple in case you guys need a simple way to reproduce the issue.

rolandshoemaker commented 2 years ago

This is an unfortunate side effect of switching to the platform verifier on macOS. For 1.20 we should probably provide a way for the TLS stack to pass through non-embedded SCTs to Verify so we can hand them to the macOS API (the windows API may also support this? but idk if they actually do anything with them for now.)

Side note: @rs this doesn't happen when using a custom root pool because when using a non-x509.SystemCertPool() pool, the Go verifier is used, which doesn't care about SCTs at all.

rolandshoemaker commented 2 years ago

In terms of a short term workaround, if you want to use all of the system roots, but there is some specific set of chains that won't work which have a set of roots you know, you can create a x509.CertPool using x509.SystemCertPool() and then explicitly add those roots to the pool using pool.AddCert(...). Pools constructed this way with use both the platform verifier (using only the system roots) and the Go verifier (using only the additionally added roots), which will bypass this particular issue. Of course you could also just use a regular x509.CertPool with just the troublesome roots, if you only care about these particular chains.

rs commented 2 years ago

Our own client based on Go is not affected because we use a custom certpool already, but there are various DoH and DoT clients based on Go out there (like dnscrypt) which are failing with such cert. We can't reasonably ask them to use a custom certpool just for our cert.

rolandshoemaker commented 2 years ago

Yup, that makes sense, just wanted to provide a viable workaround for people who do control their impl. I think the basic 1.20 fix will make this transparently disappear for most people (I will open a new issue for this, since it'll require API changes.)

isnotnick commented 2 years ago

To add: we (Sectigo) will be making a change to include SCTs even if the TLS Feature extension is present, hopefully helping to workaround this issue. @rs - @robstradling has told me the change should be implemented in the next day or so, and you should also hear from the Support team confirming the same.

rs commented 2 years ago

Thank you @isnotnick

robstradling commented 2 years ago

@rs We just implemented the change that @isnotnick described across all public ACME servers powered by Sectigo (which includes acme.zerossl.com).

thefallentree commented 2 years ago

Hello:

i can confirm that Amazon Neptune is affected :

 {"error": "x509: "*.xxxxxx.us-east-1.neptune.amazonaws.com" certificate is not standards compliant"}

what is the current recommend workaround?

melaraj2 commented 2 years ago

Amazon Elastic Cache Redis is affected as well. x509: “*.testredis.3myxt2.use1.cache.amazonaws.com” certificate is not standards compliant

mkoese commented 2 years ago

OpenShift Client 4.11.4 is affected as well.

oc login --token=sha256~xxx --server=https://api.xx.com
error: x509: “api.xx.com” certificate is not standards compliant

I just downgraded the OpenShift Client to 4.10.22 now it is working again.

ropnop commented 2 years ago

Commenting to follow along - I started seeing CMS/PKS7 signatures failing on Darwin amd64 that use Digicert's RFC3161 Timestamp Server (http://timestamp.digicert.com)

error validating signature error=invalid signature for "test.txt": x509: “DigiCert Timestamp 2022 - 2” certificate is not standards compliant

I was able to manually work around by fetching the Digicert TS Crt from https://knowledge.digicert.com/content/dam/digicertknowledgebase/DigiCertTrustedRootG4.crt and adding it to an x509.Pool to pass in for verification

AaronFriel commented 1 year ago

Pulumi had an internal user encounter this issue making requests to get.pulumi.com, served by Amazon CloudFront. The certificate for CloudFront had recently been rotated on Friday, Dec 9. They observed the issue on macOS 11.6, and upon restarting and updating to 11.7, the issue appeared to be resolved.

We're still keeping an eye on this. Has anyone else seen this behavior on Big Sur (11.x) and resolved by a patch release between 11.6 and 11.7?

Alternatively, does anyone know if the updates to the macOS system are resolving the issue, or is the update clearing some cached state that would cause it to reoccur on the next certificate rotation?

davidwallacejackson commented 1 year ago

Pulumi had an internal user encounter this issue making requests to get.pulumi.com, served by Amazon CloudFront. The certificate for CloudFront had recently been rotated on Friday, Dec 9. They observed the issue on macOS 11.6, and upon restarting and updating to 11.7, the issue appeared to be resolved.

We're still keeping an eye on this. Has anyone else seen this behavior on Big Sur (11.x) and resolved by a patch release between 11.6 and 11.7?

Alternatively, does anyone know if the updates to the macOS system are resolving the issue, or is the update clearing some cached state that would cause it to reoccur on the next certificate rotation?

If this is helpful, I just saw this error trying to connect to a GKE cluster from MacOS 12.3.1, so I assume it was clearing the cache that did it. Incidentally, if there's a way to manually do this without updating the OS, it'd be useful to know as a workaround!

stravag commented 1 year ago

For anyone struggling with this issue. I had the same problem:

package main

import (
    "log"
    "net/http"
    "os"
)

func main() {
    res, err := http.Get("https://charts.bitnami.com/bitnami/index.yaml")
    if err != nil {
        log.Fatal(err)
    }
    res.Write(os.Stdout)
}

go run <file-above>.go with version >= 1.18 lead to the mentioned certificate is not standards compliant error. It worked fine with 1.17.

Upgrading macOS Monterrey from 12.5 to 12.6.2 solved the issue for me!! It also seems to work with macOS 13 Ventura. I can now run the above program just fine with go version 1.19

jameskilroe commented 1 year ago

I am using go version go1.20.1 darwin/arm64 and MacOS Version 13.2.1 (22D68) and this issue is now occurring when I use github.com/gorilla/websocket .Dial() function.

The error is : tls: failed to verify certificate: x509: “*.exchange.coinbase.com” certificate is not standards compliant"

When is this scheduled to be resolved? Is there any simple workaround if accessing a public site where one has no control over the certificate?

Any help greatly appreciated!

AaronFriel commented 1 year ago

@jameskilroe hey! we at Pulumi dug deep into this after seeing a similar issue. You should be able to resolve it by restarting your machine. We found that should almost always fix the issue.

Workaround

We found that in every case where a user reported this issue, either of these were true:

The machine either had been recently reimaged, and not rebooted since.
The user had not restarted the machine in a substantial period of time.

In both cases, restarting the OS was the workaround.

Analysis

I'm going to summarize my colleague @kmosher's analysis. In recent years Apple, Google, and others have added the following requirements for a TLS cert to be considered valid :

It must be valid for no longer than 398 days
It must be signed by multiple Certificate Transparency (CT) logs

On macOS, and in Safari and Go programs, the system service trustd is responsible for checking a certificate against certificate transparency (CT) log information cached by the OS.

Due to unknown reasons, trustd does not update the list of trusted certificate transparency (CT) logs it uses while the system is running. As a result, certificates signed against chains that are currently trusted (and may be listed in the file below) aren't considered valid until a restart.

The trusted certificate transparency logs on macOS can be located here:

/System/Library/Security/Certificates.bundle/Contents/Resources/TrustedCTLogs.plist

And Apple publishes a JSON document for the OS to update from here:

https://valid.apple.com/ct/log_list/current_log_list.json

AaronFriel commented 1 year ago

@rsc, @ianlancetaylor: Based on the above, I believe this issue can be closed - or perhaps moved to a Discussion with an accepted answer. I believe that someone should file a report with Apple's bug reporting tool Feedback Assistant to resolve this behavior with macOS if it hasn't been. This issue may already be fixed and released, but as we describe above, users who encounter the issue are likely to be on out of date machines that have not updated or even restarted their machine recently.

jameskilroe commented 1 year ago

Hi @AaronFriel

Thanks for the answer. I updated my machine to the latest OS to try and solve the problem and did restart my machine a few time this morning, but unfortunately, the problem persists.

I did notice that the files in my TrustedCTLogs.plist were last updated on 9 Feb (before my update). Is there any chance you know how to force and update? I did some googling but no obvious answers.

All round very frustrating!

jbg commented 1 year ago

@jameskilroe rather than looking at the date, diff it against the current list (https://valid.apple.com/ct/log_list/current_log_list.json). The set of valid CT logs doesn't change that often (last change 15 Jan). If you have the latest list and have rebooted recently, then you may have a different issue (like the certificate is not actually present in the CT logs — you can check this — or is valid for too long).

andresvia commented 1 year ago

I'm sad to report that restarting my M1 also didn't worked for me.

sandheepp commented 1 year ago

tls: failed to verify certificate: SecPolicyCreateSSL error: 0 I am using aws serverless offline and trying to hit an endpoint, which is also creating issues with Mac M2. This basically makes it very inconvenient for local development after the latest update from apple.

jimidle commented 1 year ago

It’s a lien issue because AWS serverless certs deliberately don’t include certain information. You have to connect with certain parameters that exclude that part of verification, but I found that is impossible in certain cases

On Fri, Sep 15, 2023 at 13:07 Sandheep P @.***> wrote:

tls: failed to verify certificate: SecPolicyCreateSSL error: 0 I am using aws serverless offline and trying to hit an endpoint, which is also creating issues with Mac M2. This basically makes it very inconvenient for local development after the latest update from apple.

— Reply to this email directly, view it on GitHub https://github.com/golang/go/issues/51991#issuecomment-1720683610, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7TMB6LXSHICCFCR3M3K3X2PO7HANCNFSM5R4DNYCQ . You are receiving this because you commented.Message ID: @.***>

golang / go