Add pre-built FIPS AMI

[stevehipwell]

What would you like to be added: I'd like to be able to use a pre-built FIPS version of the AL2 EKS AMI provided by Amazon; AFAIK the spec for this is described in this blog. I think this is required for both AMD64 & ARM64 but AMD64 support would do for now. I'd suggest the AMI name prefix of amazon-eks-fips-node-.

Why is this needed: Everyone needing FIPS nodes needs to do the exact same thing which is the definition of toil and it should be easy to automate this as part of the AMI release process.

[seanorama]

There is already the STIG option, so adding FIPS should be easy in comparison: https://aws.amazon.com/blogs/containers/building-stig-compliant-amis-for-amazon-eks/

And AWS then to make it a pre-built image.

Other Cloud providers have this by default in their k8s node images, or as a switch as part of the k8s service startup.

This is a major point of confusion and extra work for those that are required to use FIPS. Especially considering this blog post claims that EKS meets FedRAMP High, which requires FIPS: https://aws.amazon.com/about-aws/whats-new/2021/04/amazon-eks-is-now-fedramp-high-compliant/

[stevehipwell]

I just spent most of the day Terraforming out the infrastructure to build an image for each EKS version for each architecture with EC2 ImageBuild. I think this is pretty much the definition of toil. And onboarding each account which needs to use the images will just add to that.

As an aside, the lack of an enable FIPS component for AL2 in ImageBuild seems like a major oversight. As does no support for source AMIs from SSM parameters.

[seanorama]

Related:

898 Adding FIPS 140-2 Support to EKS AMI

1007 bootstrap.sh should support setting the ecr endpoint type (such as ecr-fips)

CC: @stanhu

[ghostsquad]

I just spent most of the day Terraforming out the infrastructure to build an image for each EKS version for each architecture with EC2 ImageBuild. I think this is pretty much the definition of toil. And onboarding each account which needs to use the images will just add to that.

As an aside, the lack of an enable FIPS component for AL2 in ImageBuild seems like a major oversight. As does no support for source AMIs from SSM parameters.

@stevehipwell Would you be able to share what you've done to build these images?

[stevehipwell]

@ghostsquad we used Terraform to create build infrastructure to implement the AL2 FIPS hardening from the blog post with Image Builder. We also customise the node bootstrap to default the ECR endpoint to the FIPS one. We had to do this all from scratch as there are no existing Image Builder components to do this. there also isn't an EKS AMI release trigger or support for watching SSM so we have to manually trigger the build for each new AMI. Finally distributing AMI across partitions is hard.

[stevehipwell]

Now that #1458 has been merged it'd be great to see FIPS images published by AWS so we don't have to manage custom image distribution across partitions.

CC @cartermckinnon

[WarheadsSE]

I have to second @stevehipwell here. Having these AMI available pre-made would be exponentially easier for customers wishing to operate with FIPS enabled.

[seanorama]

Now that #1458 has been merged it'd be great to see FIPS images published by AWS so we don't have to manage custom image distribution across partitions.

This will be a huge benefit.

1. We maintain custom AMIs and then deal with the region and partition copying issues.
2. Account/region level "EBS encrypt by default", is required to be enabled which further complicated the copy/sharing due to the need for KMS+policies.
3. This also gets EKS closer to being FedRAMP authorized by default, which it is not today.

[stevehipwell]

@cartermckinnon is this something which could be reevaluated in the context of AL2023?

[rajivml]

Is there a plan to support this, many of our customers are looking for this support from aws

[bryantbiggs]

@rajivml I would encourage you to follow https://github.com/bottlerocket-os/bottlerocket/issues/1667#issuecomment-1734167141 as FIPs support is added to Bottlerocket if FIPs is a requirement for your organization

[stevehipwell]

@bryantbiggs the Bottlerocket issue is relevant but it is not a replacement for this issue, Bottlerocket is great but not everyone is happy using it for all workloads. Also getting an AL2 FIPS AMI image should just be a packaging problem (I think the same goes for AL2023), while it looks like Bottlerocket is still waiting for certification (please correct me if I'm wrong and that has all been completed)?

So we're still waiting on a maintainer response. If we can't get a quick resolution for an AL2 image (which I suspect is the case) I'd be happy with working towards a pre-built AL2023 FIPS AMI to mirror Bottlerocket.

CC @cartermckinnon

[bryantbiggs]

the Bottlerocket issue is relevant but it is not a replacement for this issue

I agree, and I am not saying its a replacement. But, the direction is still valid - if your organization requires FIPs support, the Bottlerocket AMI is going to be where you can achieve that out of the box. It is squarely within the 3 main goals of what Bottlerocket provides:

Bottlerocket has three primary goals: Minimal · Safe Updates · Security Focused.

This issue is still valid, but if you talk to any of us at AWS about meeting security and regulatory compliance and what level of "out of the box" support there is in EKS, the conversation will start with Bottlerocket

Bottlerocket is great but not everyone is happy using it for all workloads

Our Bottlerocket team would love to learn more - please let your AWS account teams know and they can get a meeting setup to chat so that the Bottlerocket experience can be improved

[stevehipwell]

@bryantbiggs Bottlerocket covers most bases, and my recommendation to our users is that it should be their default. However issues like Bottlerocket being behind even AL2 on Containerd version (1.6.x vs 1.7.x) makes this conversation harder than it should be.

[rajivml]

@bryantbiggs are there any instructions on how to enable fips with bottlerocket Linux or is there a bottlerocket linux image with fips enabled available somewhere?

[bryantbiggs]

@rajivml please see https://github.com/bottlerocket-os/bottlerocket/issues/1667 for details on FIPs support with Bottlerocket