search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.33k stars 17.7k forks source link

cmd/go: add fips140 module selection mechanism #70200

Open rsc opened 3 weeks ago

rsc commented 3 weeks ago

[!IMPORTANT] Nov 20, 2024: The latest version of the proposal is here.

Update, Nov 11 2024: An updated version is at https://github.com/golang/go/issues/70200#issuecomment-2468562595. The change is to stop presenting the choice as anything like a module, and instead to use a GOFIPS140 environment variable, analogous to GOOS, GOARCH, CGO_ENABLED, and so on.

We are working toward getting Go crypto FIPS validated, to be able to remove BoringCrypto. The relevant code is going to be crypto/internal/fips/.... People are going to need to be able to select between a few different crypto/internal/fips trees when using any given Go distribution. We propose to treat the crypto/internal/fips tree as its own pseudo-module, with a new go build flag, -fips140, to choose the version.

In general, any build needs to choose between the actual latest source in $GOROOT/src/crypto/internal/fips and an earlier source snapshot stored in module zip form in $GOROOT/lib/fips140. (Earlier snapshots will have been through more validation processes and may be necessary for certain government-related use.)

The crypto/internal/fips module will use these version numbers:

Note that the versions order correctly: v1.24.0-fips.1 < v1.24.0. The idea is that toward the end of the 1.24.0 cycle, a snapshot would be taken and sent to a lab for validation, so that by the time 1.24.0 is released, "v1.24.0-fips.1" will be an "in-process" FIPS module, meaning a lab has validated it but NIST has not yet signed off. (When NIST signs off, it becomes a "certified" module.)

The -fips140 flag can take one of the following possible values:

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of -fips140=certified in an older Go release.)

crypto/internal/fips will be like a module in the sense that:

crypto/internal/fips will be special, or unlike a module, in these ways:

The specialness seems unavoidable. We could go all the way and make fips completely different and not look anything like a module, but it is necessary to be able to say what version of fips a given program uses, and modules are the vocabulary we use for talking about versions. So it makes sense to reuse that vocabulary. In general the special cases seem like they will not cause much trouble. In particular, programs that use x/tools/go/packages or go list to find information about packages in a build will see a Dir set to a directory in the module cache when the alternate fips locations are being used. Since they already handle seeing other packages in the module cache, they should keep working without any changes.

The only special case that might cause trouble is if clients of module.CheckPath try to check "crypto/internal/fips". If that turns out to be a problem, we can update module.CheckPath to allow crypto/internal/fips as a special case. In my prototype of this functionality, I skipped the calls to module.CheckPath in the few places where it mattered, avoiding any changes to x/mod.

I have a prototype of this code working. It needs some cleanup and tests, but the necessary changes are quite small:

 src/cmd/go/internal/cfg/cfg.go               |   1 +
 src/cmd/go/internal/load/pkg.go              |   5 +-
 src/cmd/go/internal/modfetch/cache.go        |  11 +-
 src/cmd/go/internal/modfetch/codehost/git.go |   2 +
 src/cmd/go/internal/modfetch/coderepo.go     |  14 +-
 src/cmd/go/internal/modfetch/fetch.go        |   2 +-
 src/cmd/go/internal/modfetch/fips.go         | 239 +++++++++++++++++++++++++++
 src/cmd/go/internal/modfetch/repo.go         |  11 +-
 src/cmd/go/internal/modfetch/sumdb.go        |   4 +
 src/cmd/go/internal/modload/build.go         |  10 +-
 src/cmd/go/internal/modload/import.go        |  10 +-
 src/cmd/go/internal/modload/init.go          |   8 +
 src/cmd/go/internal/modload/load.go          |   2 +-
 src/cmd/go/internal/modload/query.go         |   2 +-
 src/cmd/go/internal/work/build.go            |   7 +-
 15 files changed, 306 insertions(+), 22 deletions(-)

In summary, the proposal is to establish the crypto/internal/fips pseudo-module with the version scheme and properties described above, and to add the -fips140 go build flag. (Note that “build flags” apply not just to go build but to most other go commands, including go install, go test, and go list.)

gabyhelp commented 3 weeks ago

Related Issues and Documentation

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

prattmic commented 3 weeks ago

-fips140=certified means to use the version listed in $GOROOT/lib/fips140/certified.txt (it too will be v1.X.Y-fips.N for some X, Y, N).

Am I understanding correctly that if v1.24.0-fips.1 is "in-process" when 1.24.0 is released, but certified before 1.24.1 is released, then 1.24.1 will include a backport CL to change $GOROOT/lib/fips140/certified.txt to v1.24.0-fips.1?

prattmic commented 3 weeks ago

How many "earlier snapshots" are kept in $GOROOT/lib/fips140? e.g., will I be able to use v1.24.0-fips.1 with Go 1.30.0?

I wonder mainly due to thoughts of (a) distribution size with keeping many versions and (b) there may be a maintenance burden with keeping compatibility with very old modules, particularly if there are dependencies on internals in std.

mateusz834 commented 3 weeks ago

What is the use-case for -fips140=inprocess? Why someone would need to use it instead of latest/certified. Is there any benefit/requirement for using a in process FIPS module?

mateusz834 commented 3 weeks ago

I also wonder how a process of adding any new public API to the crypto packages would look like after this change. Before adding any new API we would have to wait for it to be available in N amount of latest fips modules (https://github.com/golang/go/issues/70200#issuecomment-2457623250)? Or is the -fips140 flag going to define implicitly a build tag such that it can be detected in crypto packages for conditional compilation with //go:build.

cpu commented 3 weeks ago

What is the use-case for -fips140=inprocess? Why someone would need to use it instead of latest/certified. Is there any benefit/requirement for using a in process FIPS module?

AIUI submissions to the FIPS validation queue take a long time to become certified with the current NIST backlog. Some policy regimes (I think notably fedramp) may allow using modules that are in the process of becoming certified since they've had lab scrutiny if not yet final sign off. If a dependent user's requirements allow it then -fips140=inprocess would let you benefit from improvements made since the last certification process ended without blocking on NIST to finish the approval of the new module. Stricter policy regimes may not allow this so it has to be a separate option.

ianlancetaylor commented 3 weeks ago
  1. At this point there are no FIPS versions in process or certified. So presumably -fips140=inprocess and -fips140=certified will fail.
  2. You say that go list or x/tools/go/packages will see module information for crypto/internal/fips. For go list that will presumably be based on the -fips140 command line flag. How will x/tools/go/packages determine the FIPS version to use?
  3. I imagine that organizations will want to ensure consistent use of a FIPS version across their builds. Today they can get boringcrypto with a single consistent environment variable GOEXPERIMENT. I guess the corresponding environment variable would be GOFLAGS=-fips140=certified. But how will that be applied to x/tools/go/packages? Or (perhaps the same question) for Bazel or other build systems?
  4. It sounds like the certification process is going to be based on a particular version of the source code. What will be the schedule for further certifications? How will this affect ordinary maintenance of the crypto/internal/fips packages? Should we do typo fixes for comments? Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?
  5. When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1? (That is, can 1.25 use FIPS for 1.24.) When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1? (That is, can 1.25.3 use FIPS for 1.25.1.)
FiloSottile commented 3 weeks ago

How many "earlier snapshots" are kept in $GOROOT/lib/fips140? e.g., will I be able to use v1.24.0-fips.1 with Go 1.30.0?

I wonder mainly due to thoughts of (a) distribution size with keeping many versions and (b) there may be a maintenance burden with keeping compatibility with very old modules, particularly if there are dependencies on internals in std.

The idea is to keep only the latest certified and the latest in-process modules as snapshots. Sometimes we might need to have a third snapshot that is going to become in-process but isn't yet, but hopefully we can time things well enough that won't happen.

Indeed, the contact surface between the standard library and the modules is a burden in both directions: the modules might use internal APIs (although I tried to minimize that) and the standard library uses the module APIs (see below).

I also wonder how a process of adding any new public API to the crypto packages would look like after this change. Before adding any new API we would have to wait for it to be available in N amount of latest fips modules (#70200 (comment))? Or is the -fips140 flag going to define implicitly a build tag such that it can be detected in crypto packages for conditional compilation with //go:build.

We discussed this in abstract and we're happy with new APIs returning an error if -fips140 is not off or latest. Concretely, we haven't discussed how to implement that, but an implicit build tag is an option. As long as we're confident we can make it work, we don't need to decide the specifics until we add new module-dependent APIs.

  1. At this point there are no FIPS versions in process or certified. So presumably -fips140=inprocess and -fips140=certified will fail.

Indeed.

  1. It sounds like the certification process is going to be based on a particular version of the source code. What will be the schedule for further certifications? How will this affect ordinary maintenance of the crypto/internal/fips packages? Should we do typo fixes for comments? Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?

We'll maintain the crypto/internal/fips tree as usual, with an eye to not breaking the stdlib-facing API too often (see above). At least every year we'll kick off a revalidation to keep a healthy pipeline of module versions moving to in-process and certified.

  1. When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1? (That is, can 1.25 use FIPS for 1.24.) When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1? (That is, can 1.25.3 use FIPS for 1.25.1.)

Yes. Consider that modules take up to a couple years to reach certification at the moment, so the latest certified for Go 1.26 might still be v1.24.1-fips.1. However, only the versions that are snapshotted in lib/fips140 will be available, of which we'll only keep up to three usually (see above).

(Left out the x/tools/go/packages questions that I might not be the best person to answer.)

mateusz834 commented 3 weeks ago

We discussed this in abstract and we're happy with new APIs returning an error if -fips140 is not off or latest. Concretely, we haven't discussed how to implement that, but an implicit build tag is an option. As long as we're confident we can make it work, we don't need to decide the specifics until we add new module-dependent APIs.

Ok, not everything can/should return an error, but i assume panic is also fine, for new APIs, to avoid build tag we can also add a version const into the fips module.

ianthehat commented 3 weeks ago

Why not double down on the pseudo module idea, and instead of having a flag at all, just use replace directives. That way you can set it in the go.mod for things that really should build in fips, or in a go.work if you want to build temporarily in fips mode without touching the go.mod replace crypto/internal/fips => crypto/internal/fips/inprocess

seankhliao commented 3 weeks ago

30241 was an accepted but unimplemented proposal for standard library vendoring, would it make sense to have the fips module work like that?

n2vi commented 3 weeks ago

Inevitably, there will come a day when the latest in-process and certified version will be found to contain a vulnerability. What -fips flag is used to update to the patched version?

In OpenBSD land, we talk of versions 7.6-release (which would correspond to your certified) and 7.6-current (which is approximately your gotip) and 7.6-stable (which is the release version with only critical security and reliability patches). So another way of asking my question is what is your name for "FIPS-stable"?

I got verbal assurance in public from Donna Dodson, back when she was in charge of all this, that NIST intends that even the most compliance-sensitive government clients should run "FIPS-stable" and not wait around for re-certification. It would be good to get this in writing from the current administration, but also plan in advance for version naming.

There ought to be some associated public logs maintained at NIST for recording such patches.

FiloSottile commented 3 weeks ago

Inevitably, there will come a day when the latest in-process and certified version will be found to contain a vulnerability. What -fips flag is used to update to the patched version?

Good point, we discussed this but forgot to mention it in the proposal. The plan if a vulnerability is found in v1.24.0-fips.1 while it's the certified module is to issue v1.24.0-fips.2 and update the certified pointer in a regular Go patch release, because indeed even the most compliance-sensistive deployment is expected to weight in favor of mitigating a security vulnerability.

We can keep the previous module, too, if nonetheless we get complaints about diverging from the certified source. This can make the number of modules in lib/fips140 grow, but we did a retrospective of the security vulnerabilities in the Go crypto packages of the last five years, and only 2-4 (depending how you count) would have had to be fixed in a FIPS module, and all of them could be mitigated from the standard library. We could also be clever and compress them one against the other, if it becomes necessary.

aclements commented 3 weeks ago

Are there any values of the -fips140 that would set the GODEBUG to enforce?

The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

You say that go list or x/tools/go/packages will see module information for crypto/internal/fips. For go list that will presumably be based on the -fips140 command line flag. How will x/tools/go/packages determine the FIPS version to use?

Ping @adonovan to comment on the interaction with go/packages here.

adonovan commented 3 weeks ago

cc: @timothy-king @findleyr for x/tools implications. Seems like most of the cruft is handled by go list:

Programs that use go list or x/tools/go/packages will see the module information for crypto/internal/fips and will see source coming from the module cache when using earlier snapshots.

The GODEBUG implications are also mostly internal to go list.

There may be tools consequences (likely minor) of these rules:

  • The name "crypto/internal/fips" does not start with a domain name element, which violates golang.org/x/mod/module.CheckPath ("missing dot in first path element").
  • In -mod=vendor builds, fips code will come from the fips places ($GOROOT or else an unpacked zip), never from the vendor directory.

Also, it will be a(nother) module with no go.mod file.

FiloSottile commented 3 weeks ago

Are there any values of the -fips140 that would set the GODEBUG to enforce?

No, I expect enforce to be less popular, and applications can opt-in with the regular GODEBUG mechanisms.

The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

We've heard from a few folks that would like to make a single build and then choose to operate it in FIPS mode or not at runtime. The benefits of GODEBUG=fips140=off are faster crypto/rand and faster key generation, because FIPS 140-3 has some pretty heavy (and not generally useful) requirements on those. Another nice property of this design is that any Go binary can be operated in FIPS mode by just switching the GODEBUG at runtime, if your compliance regime is ok with "updated" modules.

I think build flag and GODEBUG are pretty orthogonal: the former selects the module version, the latter whether the binary operates in FIPS mode. They only interact in that if you used the -fips140 build flag we assume you want a GODEBUG default of on instead of off.

derekparker commented 3 weeks ago

Are there any values of the -fips140 that would set the GODEBUG to enforce?

No, I expect enforce to be less popular, and applications can opt-in with the regular GODEBUG mechanisms.

The interaction between the -fips140 flag and the GODEBUG seem pretty complicated. Do we actually need the GODEBUG? Is there a significant need for changing that at runtime rather than compile time?

We've heard from a few folks that would like to make a single build and then choose to operate it in FIPS mode or not at runtime. The benefits of GODEBUG=fips140=off are faster crypto/rand and faster key generation, because FIPS 140-3 has some pretty heavy (and not generally useful) requirements on those. Another nice property of this design is that any Go binary can be operated in FIPS mode by just switching the GODEBUG at runtime, if your compliance regime is ok with "updated" modules.

This aspect is import to us at Red Hat for how we build and ship our binaries. Likely, we will still have to carry a patch downstream because we prefer detection to be automatic (e.g. if the host is in FIPS mode, the binary should automatically operate in FIPS mode). However, having the mechanism be chosen at runtime from the initial design will make it easier to implement and carry this specific patch on our end. Additionally, overall it simplifies building and distributing binaries with FIPS requirements such that it won't require 2 builds for each component (FIPS / non-FIPS).

aclements commented 3 weeks ago

This proposal has been added to the active column of the proposals project and will now be reviewed at the weekly proposal review meetings.

timothy-king commented 3 weeks ago

crypto/internal/fips will not be listed in go.mod or participate in module version selection; the -fips140 build flag is the only way to choose the version.

I am not sure I understand the motivation behind this. Why would this pseudo-module be so special compared to other modules? My naive take is that the decision to use fips (and which version) should be checked into a file like go.mod or go.work somewhere. Is there a compliance reason?

ianlancetaylor commented 3 weeks ago

@timothy-king Are you suggesting that the possible values of the -fips140 option should instead appear in a go.mod file? It would still be a special case, because the actual source code lives in the standard Go repo. That might mean that everything that deals with go.mod files would have to know about it. Is there a big advantage one way or the other?

@adonovan Maybe this question doesn't make sense, but if something like Bazel uses go/packages uses "go list", how do pass a -fips140 option down to "go list"?

timothy-king commented 3 weeks ago

Are you suggesting that the possible values of the -fips140 option should instead appear in a go.mod file?

I am not yet suggesting this. I am trying to understand why the pseudo-module selection is not more like a normal module selection. Why not write require crypto/internal/fips v1.24.0-fips.1 or something similar in a go.mod?

Given the proposed flag, I presume that this cannot work for some reason beyond how it is distributed/where it is fetched from. But I don't understand this reason yet.

rsc commented 3 weeks ago

I am not yet suggesting this. I am trying to understand why the pseudo-module selection is not more like a normal module selection. Why not write require crypto/internal/fips v1.24.0-fips.1 or something similar in a go.mod?

@timothy-king Because this is not the kind of requirement that makes sense for a single dependency or package to enforce. For example some people need to build Kubernetes with FIPS but most people don't. It doesn't make sense to put this requirement in Kubernetes's go.mod. Even among the people who want Kubernetes with FIPS, the specific version of the FIPS code (inprocess vs certified vs latest) that those people want will differ from context to context (mostly company to company), based on their chosen compliance levels. It's very much a build-time decision not a global dependency decision.

(Edit: Same answer to @ianthehat's question about replace directives. We really don't want to allow the ecosystem as a whole to manipulate this setting.)

rsc commented 3 weeks ago

@adonovan Maybe this question doesn't make sense, but if something like Bazel uses go/packages uses "go list", how do pass a -fips140 option down to "go list"?

@ianlancetaylor They would set packages.Config.BuildFlags to include -fips140=whatever, or they could set GOFLAGS=-fips140=whatever in Config.Env or in their own environment before invoking the tool, or they could even do go env -w GOFLAGS=-fips140=whatever.

rsc commented 3 weeks ago

@seankhliao

https://github.com/golang/go/issues/30241 was an accepted but unimplemented proposal for standard library vendoring, would it make sense to have the fips module work like that?

I don't think so. We don't want people to be forced into vendoring just to use FIPS, and vendoring answers the question of "where does the code come from during the build" but not "where does the code come from when being copied into the vendor directory?"

rsc commented 3 weeks ago

Some of @ianlancetaylor's 5 questions have been answered already, but for completeness I'll answer them all in one message here.

  1. At this point there are no FIPS versions in process or certified. So presumably -fips140=inprocess and -fips140=certified will fail.

Yes, right now. Filippo is hoping to have a snapshot checked by the lab ahead of Go 1.24.0, so that in Go 1.24.0 the -fips140=inprocess flag will work. If not, then it will work in some future patch release once the snapshot does get checked. Similarly, when at some point NIST certifies a snapshot, we can issue a patch release adding a value for certified.

  1. You say that go list or x/tools/go/packages will see module information for crypto/internal/fips. For go list that will presumably be based on the -fips140 command line flag. How will x/tools/go/packages determine the FIPS version to use?

They can use packages.Config.BuildFlags or packages.Config.Env if the tool they are using lets them set those. Otherwise they can set GOFLAGS in their environment before invoking the tool, or they can use go env -w.

  1. I imagine that organizations will want to ensure consistent use of a FIPS version across their builds. Today they can get boringcrypto with a single consistent environment variable GOEXPERIMENT. I guess the corresponding environment variable would be GOFLAGS=-fips140=certified. But how will that be applied to x/tools/go/packages? Or (perhaps the same question) for Bazel or other build systems?

For x/tools/go/packages, because it invokes go list, that environment variable will have the desired effect.

There will need to be additional Bazel work, outside the scope of this proposal, to implement something like the -fips140 flag in Bazel. Specifically, the Bazel Go rules will need to be updated to have some workspace-global config knob and then adjust where the crypto/internal/fips packages come from while building the standard library. At the moment, Filippo does not have any clients who care about Bazel, so I think we can defer that work. (For Blaze inside Google, there will be different work, but it is straightforward.)

  1. It sounds like the certification process is going to be based on a particular version of the source code. What will be the schedule for further certifications? How will this affect ordinary maintenance of the crypto/internal/fips packages? Should we do typo fixes for comments? Should we do routine updates to use new standard library functionality, such as the recent changes from sort.Sort to slices.SortFunc (for cases with special-purpose implementations of sort.Interface)?

Filippo mentioned that he plans to kick off new certifications about once a year. The process for that is to make a new snapshot zip file in $GOROOT/lib/fips140 and then send it off. The certification will only apply to that zip file, not to the actual source tree, so the kinds of changes you listed are always fine to make at any time. Those kinds of cleanups would only affect -fips140=latest builds, which are completely unvalidated except for being derived from an earlier snapshot that has been validated. (That's the goal at least; right now s/has been/will have been/ or something confusing like that.) API changes in that tree are a different matter; see next answer.

  1. When using Go 1.25, will it be possible to use -fips140=v1.24.1-fips.1? (That is, can 1.25 use FIPS for 1.24.) When using Go 1.25.3 will be possible to use -fpis140=v1.25.1-fips.1? (That is, can 1.25.3 use FIPS for 1.25.1.)

When using a particular Go toolchain, say 1.25.0, it will only be possible to use the FIPS module snapshots in that toolchain's $GOROOT/lib/fips140. This limits the amount of combinatorial testing necessary for a given toolchain: we only guarantee that a particular Go toolchain works with the FIPS snapshots it ships with. The plan is in the steady state to always ship with the latest possible certified snapshot and the latest possible in-process snapshot. We may also include a latest certified+patched snapshot and latest in-process+patched snapshot, if there are critical security fixes that should be applied to a given snapshot. (For compliance reasons, some people will be able to use those fixes while others may not.)

Needing to keep some earlier snapshots working limits the amount of API changes that can be made to the API in crypto/internal/fips that is used by the rest of the crypto subtree. But changes can be made with appropriate planning, since we don't have to keep all earlier snapshots working, just the ones we choose to keep working.

ianthehat commented 3 weeks ago

@rsc I don't think replace directives allow the ecosystem as a whole to manipulate the setting, that's why I suggested them. They allow a specific module to change the default setting for it's binaries if you build directly from that module, but it is still trivial for users to override that and it libraries can't do anything to modify users. I think it gives you all the features you really need with very few downsides and no need for any new mechanisms. It would also allow people to use implementations not shipped in the standard library if they really need. I don't see yet how picking a different fips implementation is really any different to any other "long term fork" choice that users might want to make when building a binary.

FiloSottile commented 3 weeks ago

It would also allow people to use implementations not shipped in the standard library if they really need. I don't see yet how picking a different fips implementation is really any different to any other "long term fork" choice that users might want to make when building a binary.

I see making it easy to switch FIPS modules as an anti-goal, and it's why I am happy that only the ones that ship with the toolchain in lib/fips140 would be available.

The API surface between the standard library and the FIPS module is not something I want to expose, because I want us to be able to evolve it over time (albeit slowly), or even make assumptions based on undocumented properties sometimes.

Exposing it as a user-configurable replace creates an expectation that it can be, well, replaced, no matter how we document it.

Sure, downstreams can also fork the toolchain (and we recommend that in some cases), but by then they know what they're in for, and they have the control to keep up.

rsc commented 3 weeks ago

Deciding which FIPS compliance level you are building for is in many ways analogous to deciding which operating system or architecture you are building for. A generic Go program like gofmt, say, can be built for Windows or for Linux or for FreeBSD, for x86-64 or for arm64 or for mips64. We don't put that information in a go.mod, nor do we require people building gofmt to write a special go.mod or go.work just to configure their build target.

That analogy of course prompts the question of why to use a command-line flag vs an environment variable like GOFIPS140 (analogous to GOOS and GOARCH). We could plausibly do that instead, and if we took that approach, we'd probably want it to get "baked in" to the toolchain during make.bash, so that everything built with that toolchain defaulted to that GOFIPS140 setting. The command-line flag seemed clearer since it avoids more latent configuration state in the environment or baked into a toolchain, but the environment variable is fine too. I don't feel very strongly one way or the other.

I do feel strongly that this doesn't belong anywhere near go.mod or go.work, though. It's not that kind of information.

ianthehat commented 3 weeks ago

In that case I feel strongly that exposing it as a pseudo module is the wrong choice, either it is like one, in which case it should behave like one, or it is not and we should not pretend it is.

rsc commented 3 weeks ago

The motivation for the pseudo-module is to have a "dep" line in 'go version -m' that makes govulncheck work unchanged as far as understanding whether a given vuln in this code applies to a given binary or build. If we drop that 'dep' line, we would instead need to add a new field to debug.BuildInfo (or else a new BuildInfo.Setting) that lists the FIPS code version, and then govulncheck would need to add special logic to look for and understand that version and use it when deciding whether a particular vuln is present. The vuln format itself would also need to add a separate field for the FIPS version, like it has for GOOS and GOARCH, and all tools consuming the vuln format would need to handle that specially. Govulncheck can be updated easily; other scanners will probably just have another source of false positives. We can certainly take that approach.

The changes in the go command are really the same either way; it's just a question of what info is shown to govulncheck and other tools.

ianthehat commented 3 weeks ago

Yeah, I understand, but I don't think it really helps. If we make it look like a module but not be one, then things like govulncheck need to be taught all about it anyway, because it has to understand that it is not a module that obeys normal versioning rules and it can't try to work out how to upgrade to fix etc. I think we would need to think through some examples of what it would look like to add this fips specific vulnerability information to the vulndb, I am not convinced that it all works out cleanly.

timothy-king commented 2 weeks ago

We could plausibly do that instead, and if we took that approach, we'd probably want it to get "baked in" to the toolchain during make.bash, so that everything built with that toolchain defaulted to that GOFIPS140 setting. The command-line flag seemed clearer since it avoids more latent configuration state in the environment or baked into a toolchain, but the environment variable is fine too.

I was going to chime in that in my opinion it feels like what is being proposed is an implicit go generate crypto/internal/fips that selects and unpacks the zip, this creates an std that is FIPS compliant, it then compiles with that std and the GODEBUG set, and then undoes the go generate step. I think this way of looking at it is compatible with what you are saying about the changes being "baked in" to the toolchain (- the undoing the go generate). This does suggest to me that what needs to be written down is a new field in debug.BuildInfo that std is "v1.X.Y-fips.N" and not the default one for GoVersion "v1.X.Y".

Would we ever want to allow other cases of [temporarily] patching in alternative std package implementation at the command line like this?

rsc commented 2 weeks ago

@timothy-king

This does suggest to me that what needs to be written down is a new field in debug.BuildInfo that std is "v1.X.Y-fips.N" and not the default one for GoVersion "v1.X.Y".

This is only partly correct, since it mis-versions everything outside of crypto/internal/fips. It will be common with the 'certified' setting to have things like std being v1.25.0 and crypto/internal/fips being v1.24.0-fips.1. Note 25 vs 24. Or even v1.24.3 std with v1.24.0-fips.1 crypto. There really are two different versions to track.

Would we ever want to allow other cases of [temporarily] patching in alternative std package implementation at the command line like this?

Maybe, maybe not, but we're explicitly not designing a general mechanism today. (We started down that path a while back and decided there were so few uses it would not be worth the hassle. Even back then, FIPS was the only use we could think of.)

rsc commented 2 weeks ago

Spoke to @FiloSottile at GoLab, and we are happy to take @ianthehat's suggestion and make this a setting like GOOS and GOARCH. Specifically, the new proposal is:

Govulncheck will need changes to handle the GOFIPS140 metadata, much as it already has special cases for GOOS and GOARCH.

The list of module special cases are gone; nothing will ever trip over them.

Separate from the module (non-)presentation, the one new benefit of this approach is the separate environment variable. Build containers can export GOFIPS140=inprocess and be sure that no code will accidentally change that elsewhere in the build because they want to set something unrelated in GOFLAGS. The "baking in" also allows preparing an alternate go command binary that always assumes a specific GOFIPS setting, although I expect that not to be used very often.

Edit: Updated to use vX.Y as the FIPS version. The only reason for the old v1.X.Y-fips.N was to have a version to list that ordered correctly with the Go version so that non-FIPS builds could report the Go version as the version of the crypto/internal/fips module. But now that GOFIPS is a special case we can drop that requirement. There will just be v1.0, v2.0, v3.0, etc. And if we need to issue security patches, v1.1, v1.2, and so on. It's not SemVer because there is no actual SemVer semantics here.

rsc commented 2 weeks ago

Looking at the implementation, if we're not going to use modules, then the best place to hook in will be the old vendor import rewriting. The rule will be that when GOFIPS140=v1.2, then the import crypto/internal/fips/any rewrites internally to crypto/internal/fips/v1.2/any. The sources for that package will be found in $GOMODCACHE/fips140/v1.2 (unpacked there if needed).

This rewriting will mean that the FIPS v1.2 symbols will be in the binary as crypto/internal/fips/v1.2/any.Symbol, while the "latest" symbol crypto/internal/fips/any.Symbol will not be in the binary. So a vulnerability report about code in scope for FIPS will need to list the different FIPS variants that are affected as well. But then govulncheck itself needs no updates at all, because the different versions have different symbol names.

aclements commented 2 weeks ago

@rsc, what's the documentation plan for this?

rsc commented 2 weeks ago

I was going to document GOFIPS140 at the top of make.bash and in any other places where GOOS, GOARCH, GOAMD64 are documented (I can't find exactly where GOOS and GOARCH are documented today. Do you remember?)

There should also be a go.dev/doc/fips140 or perhaps go.dev/security/fips140 that gives the bigger picture overview.

ianlancetaylor commented 2 weeks ago

GOOS and GOARCH are documented, among other places, at https://pkg.go.dev/runtime from runtime/extern.go.

ianlancetaylor commented 2 weeks ago

If I understand correctly, you are suggesting that setting GOFIPS140=off when building a program means "build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

When running make.bash, not setting GOFIPS140 at all is exactly the same as setting GOFIPS140=off.

When not running make.bash, not setting GOFIPS140 at all means to use the value used when make.bash was run.

If GOFIPS140 was not set when running make.bash, or if GOFIPS140=off was set when running make.bash, and GOFIPS140 is not set when building a program, then we build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

At run time in all cases we can set GODEBUG=fips140=on to get whatever version of fips140 was used to build.

aclements commented 2 weeks ago

The proposal committee's main sense is just that this all seems complicated. On the other hand, based on my understanding, I think this is as complicated as necessary to support the needs of FIPS. This is also, to a significant extent, an expert interface. If you have to use FIPS mode, you're going to take the time to figure out what you're doing.

aclements commented 2 weeks ago

Have all remaining concerns about this proposal been addressed?

The proposal details are in https://github.com/golang/go/issues/70200#issuecomment-2468562595 (and the meanings of the GOFIPS140 values are in https://github.com/golang/go/issues/70200#issue-2635884033). The documentation plan is in https://github.com/golang/go/issues/70200#issuecomment-2474231803.

rsc commented 2 weeks ago

@ianlancetaylor

If I understand correctly, you are suggesting that setting GOFIPS140=off when building a program means "build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

Yes, and that is the default.

When running make.bash, not setting GOFIPS140 at all is exactly the same as setting GOFIPS140=off.

Yes.

When not running make.bash, not setting GOFIPS140 at all means to use the value used when make.bash was run.

Yes.

If GOFIPS140 was not set when running make.bash, or if GOFIPS140=off was set when running make.bash, and GOFIPS140 is not set when building a program, then we build with $GOROOT/src/crypto/internal/fips and act as though the program specifies //go:debug fips140=off in the main package.

Yes.

At run time in all cases we can set GODEBUG=fips140=on to get whatever version of fips140 was used to build.

Yes. It changes various behaviors, not the cryptographic code itself. It will mean that, as an incomplete list, (1) the init-time fips code+data hashing check runs, (2) crypto/rand.Reader runs the fips randomness generator, which seeds itself from /dev/random but then does more instead of using those bytes directly, and (3) crypto/tls changes its default permitted algorithms to be from the fips allowlist.

rsc commented 2 weeks ago

@aclements

The proposal committee's main sense is just that this all seems complicated. On the other hand, based on my understanding, I think this is as complicated as necessary to support the needs of FIPS. This is also, to a significant extent, an expert interface. If you have to use FIPS mode, you're going to take the time to figure out what you're doing.

I agree with this, although I think the new approach is less complicated than the module approach; thanks to @ianthehat in particular for that pushing back on that.

ianthehat commented 2 weeks ago

I agree, I think reading the whole conversation it all feels very complicated, but the actual final solution is actually only moderately complicated, no more so than it needs to be, and much cleaner than some of the intermediate steps might have you believe. Thanks for taking in to account my point of view, I am happy with where we ended up!

gopherbot commented 1 week ago

Change https://go.dev/cl/629198 mentions this issue: cmd/go: add basic GOFIPS140 support

gopherbot commented 1 week ago

Change https://go.dev/cl/629196 mentions this issue: cmd/dist: add GOFIPS140 setting

gopherbot commented 1 week ago

Change https://go.dev/cl/629201 mentions this issue: cmd/go: add GOFIPS140 snapshot support

gopherbot commented 1 week ago

Change https://go.dev/cl/629996 mentions this issue: cmd/go: fix -changed dont print when GOFIPS140 is non-default`

aclements commented 1 week ago

Based on the discussion above, this proposal seems like a likely accept.

The proposal is to add a new GOFIPS140 target configuration environment variable that controls the source used for the crypto/internal/fips package and the default value of the fips140 GODEBUG. It can take the following possible values:

GOFIPS140 crypto/internal/fips source Default fips140 GODEBUG
off $GOROOT/src/crypto/internal/fips off
latest $GOROOT/src/crypto/internal/fips on
vX.Y $GOROOT/lib/fips140/vX.Y.zip on
inprocess Use vX.Y listed in $GOROOT/lib/fips140/inprocess.txt on
certified Use vX.Y listed in $GOROOT/lib/fips140/certified.txt on

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of GOFIPS140=certified in an older Go release.)

Like all the other target configuration variables (for example, GOOS, GOARCH, GOARM, CC, CGO_ENABLED, ...), the GOFIPS140 setting during make.bash is "baked" into the toolchain as the default setting for builds with that toolchain. If unset, the default for standard Go toolchains will be "off". The baked-in default can be overridden by setting GOFIPS140 to a non-empty value when invoking go build or related commands.

When you run go version -m or fetch a debug.BuildInfo, there will be a new BuildSetting with key “GOFIPS140” and value "off" or "devel" or else a version. (That is, instead of "inprocess" you would see the specific version that "inprocess" resolved to. For "latest" you would see the Go version if this is a release, or else "devel" if that's the Go version.)

There is no user-visible API or tooling that would refer to the crypto/internal/fips code as a module.

When you run go list, packages in crypto/internal/fips/... may have a Dir in $GOROOT/src/crypto/internal/fips/... or they may come from directories in the module cache. Either way, they will have a nil Module field, matching the rest of the standard library.

While crypto/internal/fips does have a set of versions in semver syntax, they are entirely separate from the Go module system.

The files in lib/fips140 will have an implementation-defined format. That format will coincidentally use the module zip and checksum file formats, but that is an implementation detail.

GOFIPS140 will be documented at the top of make.bash, in runtime/extern.go and in any other places where GOOS, GOARCH, GOAMD64, etc are documented. There will also be a go.dev/doc/fips140 (or perhaps go.dev/security/fips140) that gives the bigger picture overview.

aclements commented 1 day ago

No change in consensus, so accepted. 🎉 This issue now tracks the work of implementing the proposal.

The proposal is to add a new GOFIPS140 target configuration environment variable that controls the source used for the crypto/internal/fips package and the default value of the fips140 GODEBUG. It can take the following possible values:

GOFIPS140 crypto/internal/fips source Default fips140 GODEBUG
off $GOROOT/src/crypto/internal/fips off
latest $GOROOT/src/crypto/internal/fips on
vX.Y $GOROOT/lib/fips140/vX.Y.zip on
inprocess Use vX.Y listed in $GOROOT/lib/fips140/inprocess.txt on
certified Use vX.Y listed in $GOROOT/lib/fips140/certified.txt on

Note that the definitions of inprocess and certified are specific to the Go toolchain being used, to keep builds reproducible. (That is, NIST issuing a certification for a new version does not change the meaning of GOFIPS140=certified in an older Go release.)

Like all the other target configuration variables (for example, GOOS, GOARCH, GOARM, CC, CGO_ENABLED, ...), the GOFIPS140 setting during make.bash is "baked" into the toolchain as the default setting for builds with that toolchain. If unset, the default for standard Go toolchains will be "off". The baked-in default can be overridden by setting GOFIPS140 to a non-empty value when invoking go build or related commands.

When you run go version -m or fetch a debug.BuildInfo, there will be a new BuildSetting with key “GOFIPS140” and value "off" or "devel" or else a version. (That is, instead of "inprocess" you would see the specific version that "inprocess" resolved to. For "latest" you would see the Go version if this is a release, or else "devel" if that's the Go version.)

There is no user-visible API or tooling that would refer to the crypto/internal/fips code as a module.

When you run go list, packages in crypto/internal/fips/... may have a Dir in $GOROOT/src/crypto/internal/fips/... or they may come from directories in the module cache. Either way, they will have a nil Module field, matching the rest of the standard library.

While crypto/internal/fips does have a set of versions in semver syntax, they are entirely separate from the Go module system.

The files in lib/fips140 will have an implementation-defined format. That format will coincidentally use the module zip and checksum file formats, but that is an implementation detail.

GOFIPS140 will be documented at the top of make.bash, in runtime/extern.go and in any other places where GOOS, GOARCH, GOAMD64, etc are documented. There will also be a go.dev/doc/fips140 (or perhaps go.dev/security/fips140) that gives the bigger picture overview.