proposal: x/crypto/x509roots/fallback: export certificate bundle

stapelberg commented 6 days ago

Proposal Details

The x/crypto/x509roots package was added in https://github.com/golang/go/issues/43958 and https://github.com/golang/go/issues/57792 (cc @rolandshoemaker @rsc @FiloSottile who were involved in prior discussion).

One feature was discussed in issue https://github.com/golang/go/issues/43958 but did not make it into the package as currently released: accessing the x509 fallback root certificate bundle programmatically, i.e. not just using it for verification in the current process, but e.g. exporting it to a file for later usage by a different process on a different machine.

Motivation / use-case

In my case, the gokrazy packer (a Go program) creates a self-contained root file system image to be run (with the Linux kernel) on a Raspberry Pi (or similar), PC or (Cloud or on-prem) VM that only contains other Go programs. A gokrazy root file system contains no C runtime or similar — similar to FROM scratch Docker containers.

The gokrazy packer can easily be run on Linux, where we just copy the system root file into the resulting image. But on macOS and Windows, we don’t have a system root file that we can copy. That’s where we currently use github.com/breml/rootcerts.

Ideally, we would programmatically create a roots file (at “gokrazy packer time”) that the Go runtime would then load (at “Raspberry Pi run time”).

Background: Why is the `x/crypto/x509roots/nss` parser not sufficient?

One might wonder: Why is the x/crypto/x509roots/nss parser not sufficient for this use-case?

I originally thought that using nss.Parse might actually make implementing breml/rootcerts easier, but it turns out that breml/rootcerts already uses an approach that does not require nss.Parse: https://github.com/breml/rootcerts/blob/7000414306b0b352acb0de167dc22ebe5a584085/generate_data.go#L34

I considered doing the http.Get in the gokrazy packer, but then my program requires internet access (undesirable, especially when running in isolated CI/CD environments) and can fail when the source is slow or unavailable. So, I’ll need a cached copy and then have to deal with keeping it up-to-date.

Instead of dealing with cache management on my user’s disk, it might be better to obtain the root certs at go:generate time and embed them into my application. But then I’m effectively doing myself the work that breml/rootcerts is currently doing for me, and have not gained anything.

I think the key observation is: obtaining the root certs is not the tricky part, but updating/distributing the root certs is an annoying problem to solve. If I could just access the fallback store that the x/crypto module already contains, GitHub’s dependabot would from time to time submit a PR to update the x/crypto dependency, and that would be the easiest solution in terms of how much infrastructure I would need to maintain.

Proposal

Add the following code to x509roots/fallback/fallback.go:

// Bundle returns the fallback X.509 trusted roots as a certificate bundle.
//
// This function is primarily useful for programs that build environments in
// which Go programs should have access to the fallback roots, such as Docker
// containers.
func Bundle() []*x509.Certificate {
    return bundle
}

https://go.dev/cl/506840 is an implementation of this proposal.

Open Questions

One open question that came up during the review of https://go.dev/cl/506840:

I'm not sure "Bundle() []*x509.Certificate" is the right API for this, for example because of https://go.dev/issue/57178: in the future the bundle in fallback might include constrained roots.

We could rename to UnconstrainedRoots(). Are constrained roots going to be vital to have a functioning certificate store?

(The whole subject of constrained roots is something I’ll need to consider when working with the generator, too, which makes it less appealing to integrate at the generator level.)

gabyhelp commented 6 days ago

Related Issues and Documentation

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

rolandshoemaker commented 5 days ago

Ah okay, so the main use case here (at least for you) is to essentially extract the parsed NSS list but without the need to fetch it yourself, and/or manage the updating of the list, and not use it in Go itself for verification, but write it out to disk (or to an image etc). This makes sense and is mostly reasonable.

One of the reasons we didn't do this as part of the initial implementation was that it introduces some slightly complicated properties that allow for mistake making. In particular if we returned a slice of pointers, we have the problem that the user can now, inadvertently or not, mutate things within the pool itself, which I'm not sure we really want to allow/encourage. We could return values instead of pointers, but that's going to be quite expensive, since these are not small objects. Perhaps that is fine though, if this is expected to be run relatively infrequently (although that clearly constrains the value of this API).

As for constraints, we now have the constraints API (although we don't use it, and just skip anything that is currently constrained). We could just return the x509.Certificate, func(x509.Certificate) error pair, and let users decide what to do themselves, or just exclude constrained certificates from the exported bundle. I think it really depends on what we expect the use cases for this API are.

stapelberg commented 5 days ago

Ah okay, so the main use case here (at least for you) is to essentially extract the parsed NSS list but without the need to fetch it yourself, and/or manage the updating of the list, and not use it in Go itself for verification, but write it out to disk (or to an image etc). This makes sense and is mostly reasonable.

Yes, that’s exactly right :)

As for constraints, we now have the constraints API (although we don't use it, and just skip anything that is currently constrained). We could just return the x509.Certificate, func(x509.Certificate) error pair, and let users decide what to do themselves, or just exclude constrained certificates from the exported bundle. I think it really depends on what we expect the use cases for this API are.

As discussed in person this morning, likely it would be good enough to just skip certificates that are constrained for now, as there are very few that have any constraints at all.

Also, we wondered whether the API should provide x509.Certificates at all, or just the unparsed NSS list as is currently stored in bundle.go (const pemRoots). The latter is sufficient for writing out the file to disk and more efficient to implement as well (no need to worry about modification of parsed x509.Certificates).

The new API could then be:

func PEMRoots() []byte {
    return []byte(pemRoots)
}

golang / go