crypto/x509: add SetFallbackRoots and golang.org/x/crypto/x509roots/fallback package

breml commented 3 years ago

Proposal

Go programs currently rely on the operating system to provide CA root certificate information in some sort of certificate system store. There are situations, where no such up-to-date CA root certificates are available like (Docker) containers built FROM scratch or out of date environments like poorly maintained or no longer updatable systems (e.g. older hardware appliances).

For some environments it is possible for the user of a Go program to add additional CA root certificates via SSL_CERT_FILE or SSL_CERT_DIR, but this is not the case for all supported environments (e.g. Windows) and it is definitively not user friendly.

Therefore, it is desirable for Go programs to have some mechanism to directly embed CA root certificate information into the program itself, so that they don't have to rely on system store to provide CA root certificates that may be absent (or out of date).

I make the following assumptions, which I think are reasonable:

adding the CA root certificates into a program should be optional, as it increases the size of the program by approximately 250K (uncompressed), and most programs run in properly maintained systems with an up-to-date certificate system store
it should be possible to embed CA root certificates when building an arbitrary program
it should be possible for a program to always embed CA root certificates without requiring any special build step
the program should always prefer data from the system if available, as it is likely to be more up to date than that included in the program
the program should allow to override the preference and force the usage of the embeded data (controled by an environment variable, e.g. GO_ROOTCERTS_ENABLE)

Given those assumptions, I propose adding a new package crypto/x509/rootcerts. Importing this package (as import _ "crypto/x509/rootcerts") will cause CA root certificates to be embedded in the program. Also, building with the build tag rootcerts will force the package to be imported, and therefore will cause CA root certificates to be embedded in the program. The embedded CA root certificates will be used if and when no certificate system store is available (or the user forces the usage of the embedded data).

Source for the CA Root Certificates

I propose to use the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in Mozilla's Root Store with the Websites (TLS/SSL) Trust Bit Enabled as the source for the CA root certificates.

The Mozilla Included CA Certificate List is the source for the CA root certificates embeded in the well known products of the Mozilla Foundation like for example Firefox (web browser) or Thunderbird (email client).

In contrast to most of the other software vendors, Mozilla maintains its Included CA Certificate List publicly and distributes it under an open source license (Mozilla Public License Version 2). This is also the reason why most of the Linux distributions, as well as other free unix derivates and wide spread tools, use this list of CA root certificates as part of their distribution.

Some examples:

Debian (and its derivates): ca-certificates
Red Hat / Fedora / CentOS: ca-certificates / ca-certificates
Alpine Linux: ca-certificates
FreeBSD: ca_root_nss
NetBSD: ca-certificates
curl: cacert.pem

In summary in my opinion it is safe to say that the Mozilla Included CA Certificate List is well established and widely used. In fact, if a Go program is run on Linux or an other free Unix derivate, chances are high that the root certificates used by the program are already provided by the Mozilla Included CA Certificate List.

Why include into the Go Standard Library

As the sample implementation (link in Annex below) clearly demostrates, that it is possible to write a 3rd party Go package, which achieves the same goal as the proposed package crypto/x509/rootcerts would. The main difference between a package in the standard library and a 3rd party package is: TRUST.

The root certificates are the top-most certificates in the trust chain and used to ensure the trustworthiness of the certificates signed by them either directly (intermediate certificates) or indirectly (through intermediate certificates). Therefore for a package containing and replacing the root certificates, trust is essential.

The same way, most users of Linux trust the CA root certificates provided by their distribution, it is very likely, that user would trust the CA root certificates provided by a package included in the Go standard library.

Additionally, the possibility to include the CA root certificates during build time, without altering the source code, is not possible with a 3rd party package but only if this package is included into the Go standard library and the build tag is implemented in to Go tool chain.

Update of the CA Root Certificates

The CA Root Certificates included in the standard library are updated with every release of Go (with the current schedule every 6 months). This would work the same way as it currently does for the package time/tzdata. The update frequency of the Included CA Certificate List is roughly every few months (2020: 5 times, 2019: 4 times, according to curl ca extract), which seems to be similar to the update frequency of the time zone data information.

In regards to updating the CA root certificates compiled into a Go binary, the same limitations apply as for the time/tzdata package. The information compiled into a binary is not updated. That being said, for the situations, this package is intended for, it is still an improvement because containers built FROM scratch are also not updated by default and out of date / not updatable systems obviously also do not get updates for the CA root certificates.

Annex

There is a sample implementation of this approach at github.com/breml/rootcerts with some additional reasoning about when to use such a package and what to keep in mind.

This proposal as well as the sample implementation are highly influenced by the proposal #38017 - time/tzdata and the implementation of the package time/tzdata by @ianlancetaylor

cc: @FiloSottile, @katiehockman, @mvdan

FiloSottile commented 2 years ago

I agree on everything, but I don't see a reasonable solution, and it doesn't feel like a reason enough not to do this at all.

I'm more hopeful that govulncheck (or vulndb entries through other scanners) will run on deployed binaries. After all, running a binary with a security vulnerability is also a major issue with a similar shape.

breml commented 2 years ago

... but a big use case for this is going to be set-and-forget binaries which once deployed are unlikely to be scanned ...

As a user of the new golang.org/x/rootcerts/bundle I like to have save defaults. But at the same time I prefer to have the freedom to handle exceptional cases the way it fits my use case. If we would always go for the save option only, we would not have InsecureSkipVerify in tls.Config. We all agree, that the default for tls must be to verify, but there are cases, where InsecureSkipVerify is needed.

In my opinion, a similar thing can be applied here:

The default behavior of golang.org/x/rootcerts/bundle could infact be to panic (or to print a warning on stderr) if the bundle is considered outdated. But as a user of this package, I would like to decide on my own, what should happen, if the bundle is considered outdated. In one case, I might decide, that the default is the right thing, because it is a (throw away) cli tool, in an other case I might decide, that I want to print a log (with the logger used in the application) and in a third case I might consider to expose the age of the bundle as a metric, such that my monitoring can warn me.

So what I have in mind here is some sort of "out of date handler", that is called if the bundle is considered out of date. The default handler would panic, but I am free to overwrite it with whatever I like.

The default behavior of panicing when the package is considered outdated obviously needs to be documented. For the ignorant user, this might lead to panics, but then I think it will raise the right questions in order to prevent this to happen again in the future. If the ignorant user decides to overwrite the "out of date handler" with a no-op, then so shall it be.

In order for this to work, the check if the bundle is recent enough would need to happen upon first usage of the fallback bundle. This would allow the user to import the package with

import _ "golang.org/x/rootcerts/bundle"

and then to overwrite the handler func with something like

bundle.OutOfDateHandler = func() {
    panic("I still prefer to panic, but with my own message")
}

If all of this is too much complexity, then I agree with @FiloSottile, that the "out of date" problematic is not reason enough to not do this at all and that govulncheck is the best option we have.

And last but not least, I would love to have SSL_ROOT_FALLBACK as mentioned in https://github.com/golang/go/issues/43958#issuecomment-1283094137.

breml commented 2 years ago

With all of that said, one problem with long running services remains anyway, because as far as I know, the system roots are not reloaded from time to time by x509. So even if I update the roots on my machine, a long running service will continue with the old roots loaded when the application was started.

rsc commented 2 years ago

How are we doing on this discussion? Do people agree with the most recent API proposal in https://github.com/golang/go/issues/43958#issuecomment-1284014369 ?

stapelberg commented 2 years ago

How are we doing on this discussion? Do people agree with the most recent API proposal in #43958 (comment) ?

Yes, the proposed rootcerts.FetchRootsAsPEM API would work for me AFAICT :)

rolandshoemaker commented 2 years ago

Yup, I think this is pretty much what we've settled on.

breml commented 2 years ago

Yes, I agree with https://github.com/golang/go/issues/43958#issuecomment-1284014369.

The only thing that I still request to include is the support for SSL_ROOT_FALLBACK environment variable to force the usage of the embedded root certificates (e.g. on a system with outdated root certificates).

rsc commented 1 year ago

Is SSL_ROOT_FALLBACK a standard environment variable in other non-Go systems? If not, we should probably use a variable with a GO prefix, and probably just a setting in the GODEBUG variable.

rsc commented 1 year ago

I am not sure that golang.org/x/rootcerts/bundle is the right import path.

First, I don't think we want to maintain a whole repo just for root certs. Probably we should use an x/crypto submodule if we need to separate them.

Second, for a package that is imported purely for side effects, "bundle" does not really indicate what those side effects are.

Since the role of the package is to register with x509.SetFallbackRoots, perhaps the package should be

import _ "golang.org/x/crypto/x509roots/fallback"

stapelberg commented 1 year ago

I like the fallback name! Definitely more descriptive than bundle :)

breml commented 1 year ago

Is SSL_ROOT_FALLBACK a standard environment variable in other non-Go systems? If not, we should probably use a variable with a GO prefix, and probably just a setting in the GODEBUG variable.

No, SSL_ROOT_FALLBACK is not a standard environment variable as far as I know. For me, a GODEBUG variable is fine as well, as long as the forced usage of the fallback root certs can be controlled from the outside (at start time of the application).

First, I don't think we want to maintain a whole repo just for root certs. Probably we should use an x/crypto submodule if we need to separate them.

For me the important part is not about having separate repositories (this can be decided based on e.g. maintenance effort per repo), the important thing is, that want to be able to version the golang.org/x/crypto/x509roots independently from golang.org/x/crypto in order to leverage govulncheck and dependabot whenever there is a new set of roots available. It is my understanding, that this is perfectly possible with submodules and hierarchical git tags. So for me, golang.org/x/crypto/x509roots for the submodule and golang.org/x/crypto/x509roots/fallback for the automated registering of the roots is fine. Speaking of the package name, what about golang.org/x/crypto/x509/roots and golang.org/x/crypto/x509/roots/fallback? This would work as well, wouldn't it?

rsc commented 1 year ago

Sounds like other than the package name people are happy with this proposal.

rolandshoemaker commented 1 year ago

+1 for using a GODEBUG variable to force the fallback.

rsc commented 1 year ago

Based on the discussion above, this proposal seems like a likely accept. — rsc for the proposal review group

gopherbot commented 1 year ago

Change https://go.dev/cl/449235 mentions this issue: crypto/x509: implement SetFallbackRoots

FiloSottile commented 1 year ago

I don't want to get in the way of this getting accepted before the freeze, and this is a minor note, but after looking at the implementation I would rather not have the force fallback GODEBUG.

On macOS and Windows, the fallback would normally not be reachable, because the OS verifier is always there. What does x509usefallbackroots do on those platforms? Does it make the fallback available anyway?

What happens if x509usefallbackroots is set but SetFallbackRoots was not called? An error? A panic? Is it ignored?

SSL_CERT_FILE=/dev/null SSL_CERT_DIR=/dev/null is certainly not the prettiest, but at least it has very clear semantics: it only applies to Linux because SSL_CERT_FILE/DIR do, and if there are no fallback roots you get an error.

breml commented 1 year ago

On macOS and Windows, the fallback would normally not be reachable, because the OS verifier is always there. What does x509usefallbackroots do on those platforms? Does it make the fallback available anyway?

In my opinion, the GODEBUG variable should force the usage of the embedded certificates regardless of the operating system. There exist very old installations of especially Windows systems in the wild, where the root certificates provided by the system might be outdated as well. In such a case, the GODEBUG force fallback should allow to force the respective Go application to use the embedded certificates regardless of the fact, that a Windows system always provides some set of root certificates via the OS verifier. For me, the GODEBUG option indicates that the user wants to use the embedded certificates in any case.

What happens if x509usefallbackroots is set but SetFallbackRoots was not called? An error? A panic? Is it ignored?

I am fine with the GODEBUG x509usefallbackroots getting ignored, if no fallback roots are set. If we decide to go with the panic, it is important in my opinion, that this panic is happening during initialization of the application. I don't like an application to panic only after some time, when a certificate is actually used (imaging a long running service, which occasionally queries an API via HTTPS and then only fails when the first call to the API is executed). I have not looked into the implementation, so I can not judge, if an error is a viable solution.

SSL_CERT_FILE=/dev/null SSL_CERT_DIR=/dev/null is certainly not the prettiest, but at least it has very clear semantics: it only applies to Linux because SSL_CERT_FILE/DIR do, and if there are no fallback roots you get an error.

Again, I don't think the fallback certificates should be limited to *nix OS.

rsc commented 1 year ago

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal. — rsc for the proposal review group

rolandshoemaker commented 1 year ago

I've been mostly sitting on the fence about the GODEBUG behavior, but I think we need to make a final decision and I'm going to propose the following (which is, essentially, what is implemented in my CL):

When calling SetFallbackRoots with GODEBUG= x509usefallbackroots=0, if the system pool is empty it will be replaced with the fallback pool. On Windows and macOS we will inherently never fallback, since we don't know anything about the content of the system pool, as we use platform APIs for verification (so we essentially always consider the pool non-empty.)
When calling SetFallbackRoots with GODEBUG= x509usefallbackroots=1, the system root pool will be replaced regardless of its content or the platform. On macOS and Windows this will inherently disable use of the platform verification APIs since the special system pool will be replaced with a generic pool (we could also create a hybrid pool, with would result in parallel verification both with the fallback pool, and the platform APIs, but I think this is more confusing.)
If SetFallbackRoots is not called with GODEBUG= x509usefallbackroots=1, SystemCertPool acts as if a fallback pool was never set, and behaves normally.

This gives us (mostly) uniform behavior across platforms, and is the most obvious behavior. Given this is a GODEBUG flag, even though I'm sure people will end up relying on the semantics of this behavior in strange unexpected ways, we'll have a little bit more leeway to change it in the future in case we realize down the road this is wrong.

andig commented 1 year ago

I like the fallback name! Definitely more descriptive than bundle :)

I was gonna ask if OS certs would be used if present. That answers it- yes. I really like this change. One of hour top support issue with Docker is people mounting anything into /etc and then complaining about apparently unrelated networking errors. I assume this change should fix the situation.

pete-woods commented 1 year ago

I'm not 100% sure I follow all the discussions, but one thing in particular we (at CircleCI) would like out of this (besides the FROM scratch Docker scenario) is to force the Go based certificate store, even when os-provided certificates are present. E.g. someone runs our software in an old Docker image with an out of date set of rootcerts in.

breml commented 1 year ago

Hi @pete-woods

I'm not 100% sure I follow all the discussions, but one thing in particular we (at CircleCI) would like out of this (besides the FROM scratch Docker scenario) is to force the Go based certificate store, even when os-provided certificates are present. E.g. someone runs our software in an old Docker image with an out of date set of rootcerts in.

https://github.com/golang/go/issues/43958#issuecomment-1317868626 summarizes how this will work. So I guess, your use case is (only) partly solved. Whenever a user needs to force the Go embedded certificates (e.g. because the os-provided roots are out of date), he will need to pass the GODEBUG= x509usefallbackroots=1 environment variable. I pushed for a feature like the one you are requesting, but I had no support, so this was not included.

breml commented 1 year ago

@pete-woods Thinking of it a little bit more, there might be a way of setting the environment variable from code in an import, that is loaded before SetFallbackRoots is called (similar to how it is done in the unit tests in https://github.com/golang/go/commit/04d6aa6514617d5284f0657928eccb579a0f42e2#diff-aab8875f722653c06a57f1a4509bdad43391f1e2dc9555644a56beeaf96bb808R91).

rsc commented 1 year ago

The crypto/x509 API has landed and should be in Go 1.20. For the purposes of release notes we have to say what the supporting package will be. Roland and I talked and agreed on golang.org/x/crypto/x509roots/fallback.

andig commented 1 year ago

I'm struggelling using the fallback package (go1.20):

go get golang.org/x/crypto/x509roots/fallback@latest
go: module golang.org/x/crypto@latest found (v0.7.0), but does not contain package golang.org/x/crypto/x509roots/fallback

go get golang.org/x/crypto/x509roots/fallback@master
go: downloading golang.org/x/crypto v0.7.1-0.20230320203329-018c28f8a114
go: module golang.org/x/crypto@master found (v0.7.1-0.20230320203329-018c28f8a114), but does not contain package golang.org/x/crypto/x509roots/fallback

pete-woods commented 1 year ago

I don't think this package has actually been released yet (despite being referred to in the 1.20 release notes)

dmitshur commented 1 year ago

@rolandshoemaker With SetFallbackRoots present and documented in Go 1.20, and golang.org/x/crypto/x509roots/fallback being recently added as part of #57792, is it time to move this issue to Go1.20 milestone and close it? Or is there more that still needs to happen here?

rolandshoemaker commented 1 year ago

Yes, thanks for the reminder! 🎉

golang / go