FIPS Support / Validation

[vennemp]

This is related to my comment in #1297

Will you adding support for FIPS 140-2 standards? Please note: I am NOT asking about FIPS validation with regards to the containers themselves - I understand that is handled, at least technically speaking, during the build phase of the container image. FIPS 140-2 compliance for other OSes (Amazon Linux 2, RHEL) configures the crypto module to operate in compliance (proper crypto algorithms, entropy, etc) - these would be done in container image; however, there are other changes at the kernel-level that occur as well. I don't believe full on validation is in scope for Bottlerocket since crypto modules themselves get validated (not necessarily an OS) but perhaps guidance and/or documentation on meeting compliance requirements would be helpful.

[zmrow]

Thanks for opening this! We'll look into it.

[technotaff-nbs]

A lack of compliance statement for CIS is preventing our company (and i'm sure many others) from adopting Bottlerocket.

[JohnTrevorBurke]

Any updates around FIPS compliance and the usage of Bottlerocket OS? It's a blocker for our team to be able to use for Fedramp compliant EKS nodes. Any updates would be very much appreciated!

[vennemp]

Hey @JohnTrevorBurke i believe I just added you in linked in. I may be able to help.

[geekmuse]

In the same boat here re: FedRAMP. Any info/updates here?

[FringeCase]

Bumping @geekmuse comment; FedRAMP path

[walesb]

Hi Team, I have the same query here, I just want to confirm to know if AWS BottleRocket AMI's are FIPS-140-2 compliant by default or not?

Looking forward to your reply.

[stevehipwell]

Is there any news on this?

[bcressey]

FIPS compliance is our second most requested feature, behind CIS (which is in progress), and I'm planning to focus on it once the CIS benchmark is complete.

There are (at least) two interpretations of what this means, and different auditors may have different takes:

1. Bottlerocket's kernel uses compliant algorithms when fips=1 is passed on the kernel command line, and its userspace tries to arrange for containers to do the same (by mounting in /etc/system-fips and such).
2. In addition to the above, Bottlerocket's kernel and userspace are themselves FIPS certified, meaning that they only use validated, FIPS-compliant crypto.

The first should be somewhat lower hanging fruit, and it's how I understand @vennemp's initial request. But I've also heard interest in the second, stronger version. That's a bigger lift because it likely means switching away from rustls and Go's crypto library, using OpenSSL for everything instead, and adopting something like Go FIPS rather than the regular Go toolchain.

[bputt-e]

Is this planned to be done in early 2023?

[stmcginnis]

Hi @bputt-e - the hope is someone can work on this in 2023, but there is no definite time set at this point.

[oded-dd]

Hey. Any update on that? Same path of "Fedramp"

[voidlily]

Also looking for this for fedramp purposes

[amandagalligan]

Hi @stmcginnis wanted to check in on updates or any rough timeline for Bottlerocket FIPS 140-2 support in 2023?

[stmcginnis]

Hi @amandagalligan! No timeline yet, but FIPS compliance is an important thing on the roadmap. Unfortunately it will take some time to get everything in place. But there is work being done towards that end. I just can't give any kind of time estimate at this point, unfortunately.

[filipenf]

Would it make sense to publish an image containing only the first part mentioned by @bcressey above, since it is a low hanging fruit?

It is a good starting point for most customers - we'd still need to solve the user space library problem - which needs to be solved on a case by case basis anyway

[jpmcb]

Hi @filipenf - thanks for the comment. Like Ben mentioned, implementing fips=1 being passed to the kernel command line is pretty low hanging fruit. But publishing and distributing an image for Bottlerocket that supports that is a heavy lift on its own. And we want to make sure we're shipping the most complete and quality software we can.

Like Sean mentioned, there's work being done to make this happen since this is a very highly requested feature but there's no timeline yet.

[awilmo8]

Just a bump, we are also tracking this and it's FIPS compliance for FedRAMP purposes. Thanks for all you guys do!

[Eyalsadeh]

Any idea if someone knows when it will be ready?

[stmcginnis]

Not much of an update, but current status is there are a few things in-flight towards getting FIPS certification, but it it will be some time before that can be completed.

Bottlerocket leverages the work done with the kernel from Amazon Linux. There is active work being done there to support FIPS. Then the certification process itself takes some time to go through. So I don't have a lot of concrete details, but with the work that needs to be done yet, it is looking like it won't be any sooner than some time in 2024.

[Eyalsadeh]

Thanks for the update

[pburkholder]

Looks like AWS LC is nearing FIPS certification (https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/Modules-In-Process-List) Will that help, or do you use AWS-LC Cryptographic Module v2.0 and Amazon Linux 2023, which are much farther behind in the certification process?

[jacurtis]

Thanks for the update. I am bumping because we are yet another org that needs FIPS for FedRAMP compliance. I was just about to go live in a few weeks with an EKS cluster using bottlerocket and was told today by the compliance team that we need to switch OS on our nodes because of this. So it is a full blocker for us.

No FIPS = No FedRamp = No Bottlerocket 😢

Hoping this gets resolved soon.

[adamcrosby]

@vennemp The OS is definitely in scope if the container uses the shared kernel for anything requiring a cryptographic operation. fips=1 must be passed to the linux kernel at boot time to ensure the kernel itself only utilizes appropriate cryptographic altos/etc. Additionally, the container crypto libs (openssl/etc.) must ALSO be FIPS. Tons of drivers do things like TLS offload or acceleration, etc. as an example. If I were validating, you'd have to prove to me your app/data at no point is exposed to kernel crypto to let it be marked out of scope. The host OS user-space modules I can see easily being out of scope with a containerized app, but not the kernel.

[vennemp]

Are you aware of any container images that don't include their own cryptographic modules? If so, I would imagine their use is very limited; it could limit the container's portability between OSes and you'd have to mount a volume on the host, where the crypto libs and module reside, inside the container at runtime for it to be used. Your point is well taken though, in the unlikely event the container is using the host's modules, yes that would make the OS's FIPS status relevant for the application.

One small wrinkle I am aware of is the use of Cilium's Transparent Data Encryption to automatically encrypt node to node traffic. This uses the host's kernel implementation of IPSec. But the container that runs Cilium is not actually performing the encryption, it is making changes to the host's routing config to create the IPSec tunnels- which is a subtle but important difference. Since the host is actually encrypting the traffic then the FIPS status of the host is in scope when using Cilium's TDE. Any other application that works in a similar fashion would require FIPS support on host and would preclude you from using bottlerocket until this is addressed.

[srguglielmo]

RedHat UBI 9 containers require FIPS mode to be enabled on the host OS and require a bind-mount of /etc/system-fips from the host into the container in order for the FIPS validation tests to pass successfully. (Despite having its own crypto libs in the container).

[bcressey]

One fairly widespread example of a container dependency on host crypto is the kernel PRNG, accessible via /dev/random, /dev/urandom, or the getrandom() syscall. Pretty much every standard library or runtime for popular languages will default to these mechanisms for getting random bytes on Linux.

[bcressey]

I've owed this issue a more substantial update for a while so here goes.

In the current plan, FIPS for Bottlerocket requires two modules:

• AWS-LC Cryptographic module (in "Coordination" state)
• Amazon Linux 2023 Kernel Crypto API Cryptographic Module (in "Implementation Under Test" state)

NIST doesn't share the expected timeline but based on historical data scraped from those links, it looks like validation may take up to 650 days - meaning December 2024 before the kernel will be validated.

The remaining implementation work required is:

• Create a golang build that swaps in aws-lc for the Go crypto module, using the upstream GOEXPERIMENT=boringcrypto.
• Figure out how to plug in aws-lc-rs instead of ring at build time.
• Adjust templates and helpers to automatically use FIPS API endpoints instead of non-FIPS

Given that FIPS entails some functional and cryptographic trade-offs - no Wireguard support in the kernel, none of the newer, well-regarded non-FIPS algorithms, more overhead in crypto operations in Go, etc - my plan is to release a parallel set of FIPS variants.

For example:

• aws-k8s-1.28 would be non-FIPS, use native Rust and Go crypto, and provide the full set of kernel functionality, same as today.
• aws-k8s-1.28-fips would be FIPS enabled out of the box, use aws-lc backed crypto, have fips=1 on the kernel command line, mount /etc/system-fips into containers, and use FIPS API endpoints for communicating with EKS, ECR, and EC2.

[bcressey]

Some exciting news from last week - AWS-LC is now FIPS 140-3 certified - so it's just the 6.1 kernel that's left.

[bcressey]

The 6.1 kernel has reached the interim validation stage - so we're getting closer.

[bcressey]

For those interested in FIPS AMIs - I'm curious whether there's any use case for publishing them in regions other than the ones listed here:

• us-east-1
• us-east-2
• us-west-1
• us-west-2
• us-gov-east-1
• us-gov-west-1
• ca-central-1
• ca-west-1

@arnaldo2792 is evaluating whether to set AWS_USE_FIPS_ENDPOINT=true to force the use of FIPS endpoints, which has this property:

If this setting is enabled and a FIPS endpoint does not exist for the service in your AWS Region, the AWS call may fail.

That in turn would make the FIPS AMIs rather useless outside of regions where EC2, EKS, ECS, and ECR FIPS endpoints were available. (edit to add: this apparently includes the Canada regions also, except for EC2).

[jxn]

@arnaldo2792 is evaluating whether to set AWS_USE_FIPS_ENDPOINT=true to force the use of FIPS endpoints, which has this property:

If this setting is enabled and a FIPS endpoint does not exist for the service in your AWS Region, the AWS call may fail.

That in turn would make the FIPS AMIs rather useless outside of regions where EC2, EKS, ECS, and ECR FIPS endpoints were available.

To be honest, I have use-cases for both behaviors. There are definitely instances where, if a FIPS endpoint is not available in a region or even for a service as a whole, I strongly prefer a failure and error message to a silent failover. This, again, would be best if it works this way for both a region that does not support FIPS endpoints AND for a service that does not support it at all. (Ideally, this would be the same as AWS_USE_DUALSTACK_ENDPOINT. And, if I specify both USE_FIPS_ENDPOINT AND USE_DUALSTACK_ENDPOINT to true, I'd hope to fail with an error if said endpoint doesn't support BOTH FIPS and ipv6). AWS_REQUIRE_FIPS_ENDPOINT would seem like a fair new var if you don't want to break configurations that currently exist.

I also have a number of container resources and IAC which are only different in that one should use FIPS endpoints and the other is agnostic. For these items, it's helpful that USE_FIPS_ENDPOINT=true falls back to a non-fips endpoint, so I don't have to build two different versions for both scenarios. If both behaviors were available, this would be helpful for when I have to deal with a mix of services in the same context, some of which support FIPS and some of which do not. I'd consider this the less important, as I can work around it ... having to know every service and region with support and buildling checks into the software and IAC

[stevehipwell]

We have a requirement to run FIPS nodes in regions without FIPS ECR endpoints (e.g. Canada). I think defaulting to the FIPS ECR endpoint for the FIPS version across the board would be best, with the ability to change the ECR endpoint to non-FIPS for the regions where it's not available. This would support all use cases but make the use of a non-FIPS ECR endpoint require explicit configuration.

[vennemp]

The 6.1 kernel has reached the interim validation stage - so we're getting closer.

@bcressey Maybe I'm missing something in the Security Policy but is there a reason why AWS is being so restrictive in what it defines in the Vendor Affirmed Operational Environments for this module? I've read the security policy a couple times now, and I cannot find anything that mitigates this issue.

Check page 9 of CMVP #4808.

For Amazon Linux 2023 and Bottle Rocket you are basically requiring us to use c7g.metal or c6i.metal instances - since those are what you tested in or are affirming. I get you want to be explicit in the testing operational environment - but other Vendors typically use the Vendor affirmed Operational environment to be more open so more configurations are covered by the certificate.

For example: Boring Crypto CMVP Policy basically lists any version of Linux as an affirmed operational environment.

Trend Micro's Policy basically does the same. They are explicit in the testing environment and more open on vendor affirmed operational environments.

While I admit it is unlikely an auditor will enforce this level of granularity - they are well within their rights to mark it as a finding for not using FIPS 140-3 Validated Cryptography if the configuration does not match the policy. I've heard from other AWS employees that in internal documentation regarding previously validated modules, the modules were affirmed to work on any EC2 instance type - but it was NOT listed in the CMVP security policy it was only mentioned in internal documentation. FWIW, I believe it was this module Why omit this information from the security policy - since that is what auditors refer to if they have questions?

[ginglis13]

Hi @stevehipwell, I wanted to clarify on your suggested approach to fulfill your requirement:

We have a requirement to run FIPS nodes in regions without FIPS ECR endpoints (e.g. Canada). I think defaulting to the FIPS ECR endpoint for the FIPS version across the board would be best, with the ability to change the ECR endpoint to non-FIPS for the regions where it's not available. This would support all use cases but make the use of a non-FIPS ECR endpoint require explicit configuration.

For Bottlerocket's host containers, and host-ctr, I have opened https://github.com/bottlerocket-os/bottlerocket-core-kit/pull/204. This will add functionality to host-ctr such that it resolve FIPS ECR endpoints if FIPS is supported in the region specified by the image URI, and it will set the URIs for host containers when run in FIPS mode.

We're also planning to set use_fips_endpoint=true in AWS config via Bottlerocket's settings.aws.config by default for FIPS variants, which will address your requirement to use FIPS AWS service endpoints by default, when available.

Bottlerocket will default to the FIPS service endpoint for AWS services in FIPS variants, and be intentional about FIPS ECR endpoints in host containers / host-ctr. Will that still work for your requirements?

[stevehipwell]

@ginglis13 that looks to be a very comprehensive solution for our requirements.

[adamcrosby]

For those interested in FIPS AMIs - I'm curious whether there's any use case for publishing them in regions other than the ones listed here:

• us-east-1
• us-east-2
• us-west-1
• us-west-2
• us-gov-east-1
• us-gov-west-1
• ~ca-central-1~
• ~ca-west-1~

us-iso* and the other 'special' partitions would be super useful.