Configure FIPS AWS service endpoints

ginglis13 commented 1 week ago

Issue number: Related: https://github.com/bottlerocket-os/bottlerocket/issues/1667

Description of changes:

This PR ensures that Bottlerocket FIPS variants use FIPS AWS service endpoints by default. It adds a setting generator, aws_config, which will be used to set settings.aws.config by default if this setting is null. The value of the setting will be base64 encoded

[<settings.aws.profile>]
use_fips_endpoint=<true|false>

Changes are made to pluto to ensure that settings.aws.config is used for its AWS service calls, given that pluto runs before settings are applied. This is done by reading the inflight value of settings.aws.config, storing it to a file, /root/.aws/config.pluto, and temporarily setting AWS_CONFIG_FILE to this config file.

Testing done:

I built FIPS aws-k8s-1.29 variants in us-west-2, us-gov-west-1, ca-central-1, and eu-west-1 for testing. These AMIs include changes for

fips = true in variant definition.
settings.aws.config rendered by schnauzer, if not set, to include use_fips_endpoint=<true|false> depending on if the AMI is FIPS enabled (aka, using the aws_config helper added in this PR)

Ran manual tests for the following scenarios and outcomes with a dev aws-k8s-1.29 FIPS variant:

AMI	Region	use_fips_endpoint	Boots?	Joins Cluster?
FIPS-enabled	us-west-2	TRUE	yes	yes
FIPS-enabled	us-west-2	FALSE	yes	yes
FIPS-enabled	us-gov-west-1	TRUE	yes	yes
FIPS-enabled	us-gov-west-1	FALSE	yes	yes
FIPS-enabled	ca-central-1	TRUE	yes	no
FIPS-enabled	ca-central-1	FALSE	yes	yes
FIPS-enabled	eu-west-1	TRUE	no	no
FIPS-enabled	eu-west-1	FALSE	yes	yes

The reason for the slight difference in behavior for ca-central-1 and eu-west-2 is that ca-central-1 does have EC2 FIPS endpoints: https://aws.amazon.com/compliance/fips/#FIPS_Endpoints_by_Service

I added debug logging to pluto to ensure that use_fips for the AWS SDK it uses returns true or false depending on the /root/.aws/config.pluto that is used.

Details for the happy path, with debug logging in pluto, are below (I can include the details for all others in the above table, I just don't want to clutter this description):

FIPS enabled AMI, FIPS enabled region - settings.aws.config is set with use_fips_endpoint=true, boots successfully and joins cluster ✅

``` bash-5.1# apiclient get settings.aws { "settings": { "aws": { "config": "W2RlZmF1bHRdCnVzZV9maXBzX2VuZHBvaW50PXRydWU=", "profile": "default", "region": "us-west-2" } } } bash-5.1# journalctl -u pluto Oct 23 21:55:25 localhost systemd[1]: Starting Generate additional settings for Kubernetes... Oct 23 21:55:25 localhost settings-committer[1339]: 21:55:25 [INFO] Checking pending settings. Oct 23 21:55:25 localhost settings-committer[1339]: 21:55:25 [INFO] Committing settings. Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] pre set config AWS_CONFIG_FILE_ENV_VAR: unset Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Setting AWS config Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Wrote AWS config to "/root/.aws/config.pluto" Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] AWS_CONFIG_FILE_ENV_VAR: /root/.aws/config.pluto Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] The AWS_CONFIG_FILE is /root/.aws/config.pluto, which sdk_config should use Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs:Fs(Real), time_source: SharedTimeSource(SystemTimeSource), http_client: None, sleep_impl: Some(SharedAsyncSleep(TokioSleep)), region: None, use_fips: None, use_dual_stack: None, profile_name_override: None } } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=ImdsRegionProvider { client: "IMDS client truncated for readability", env: Env(Real) } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] sdk_config use_fips: Some(true) Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] lazy_load_identity; Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Received meta-data/local-ipv4 Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Received meta-data/instance-type Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Received meta-data/instance-id Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Received dynamic/instance-identity/document Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] Received meta-data/instance-id Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] The AWS_CONFIG_FILE is /root/.aws/config.pluto, which sdk_config should use Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=EnvironmentVariableRegionProvider { env: Env(Real) } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=ProfileFileRegionProvider { provider_config: ProviderConfig { env: Env(Real), fs:Fs(Real), time_source: SharedTimeSource(SystemTimeSource), http_client: None, sleep_impl: Some(SharedAsyncSleep(TokioSleep)), region: None, use_fips: None, use_dual_stack: None, profile_name_override: None } } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] load_region; provider=ImdsRegionProvider { client: "IMDS client truncated for readability", env: Env(Real) } Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] sdk_config use_fips: Some(true) Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] lazy_load_identity; Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] AWS_CONFIG_FILE_ENV_VAR: /root/.aws/config.pluto Oct 23 21:55:26 localhost pluto[1342]: 21:55:26 [INFO] And finally unset. AWS_CONFIG_FILE_ENV_VAR: unset Oct 23 21:55:26 localhost systemd[1]: Finished Generate additional settings for Kubernetes. bash-5.1# systemctl status kubelet ● kubelet.service - Kubelet Loaded: loaded (/x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled) Drop-In: /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/service.d └─10-security.conf /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d └─dockershim-symlink.conf /etc/systemd/system/kubelet.service.d └─exec-start.conf /x86_64-bottlerocket-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service.d └─make-kubelet-dirs.conf, prestart-load-pause-ctr.conf Active: active (running) since Wed 2024-10-23 21:55:30 UTC; 34min ago Docs: https://github.com/kubernetes/kubernetes Process: 1450 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS) Process: 1453 ExecStartPre=/bin/ln -sf ./containerd/containerd.sock /run/dockershim.sock (code=exited, status=0/SUCCESS) Process: 1456 ExecStartPre=/usr/bin/mkdir -p /var/lib/kubelet/providers/secrets-store /var/lib/kubelet/node-feature-discovery/features.d (code=exited, status=0/SUCCESS) Process: 1457 ExecStartPre=/usr/bin/ctr --namespace=k8s.io image import --all-platforms /usr/libexec/kubernetes/kubernetes-pause.tar (code=exited, status=0/SUCCESS) Process: 1478 ExecStartPre=/usr/bin/ctr --namespace=k8s.io image label localhost/kubernetes/pause:0.1.0 io.cri-containerd.pinned=pinned (code=exited, status=0/SUCCESS) Main PID: 1483 (kubelet) Tasks: 12 (limit: 4511) Memory: 222.2M CPU: 14.790s CGroup: /runtime.slice/kubelet.service └─1483 /usr/bin/kubelet --cloud-provider external --kubeconfig /etc/kubernetes/kubelet/kubeconfig --config /etc/kubernetes/kubelet/config --conta… Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.262827 1483 kubelet_node_status.go:497] "Fast updating no…e ready" Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.367284 1483 topology_manager.go:215] "Topology Admit Hand…c-q6d6l" Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.367713 1483 topology_manager.go:215] "Topology Admit Hand…c-m4qs9" Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.568657 1483 reconciler_common.go:258] "operationExecutor.VerifyCo… Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.568752 1483 reconciler_common.go:258] "operationExecutor.VerifyCo… Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.568808 1483 reconciler_common.go:258] "operationExecutor.VerifyCo… Oct 23 21:55:52 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:52.568834 1483 reconciler_common.go:258] "operationExecutor.VerifyCo… Oct 23 21:55:59 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:59.122833 1483 pod_startup_latency_tracker.go:102] "Observed pod sta… Oct 23 21:55:59 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:55:59.144183 1483 pod_startup_latency_tracker.go:102] "Observed pod sta… Oct 23 21:56:13 ip-192-168-3-174.us-west-2.compute.internal kubelet[1483]: I1023 21:56:13.183187 1483 pod_startup_latency_tracker.go:102] "Observed pod sta… Hint: Some lines were ellipsized, use -l to show in full. ```

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

ginglis13 commented 1 week ago

^ force push adds unit test for the aws_config schnauzer helper

ginglis13 commented 1 week ago

^ force pushes stores the AWS config as a tempfile, removes the env::remove of AWS_CONFIG_FILE, addresses other feedback.

ginglis13 commented 1 week ago

^ rebase

ginglis13 commented 1 week ago

^ force push includes

style changes
in the aws-config helper, only set settings.aws.config if settings.aws.profile is either "default" or null.
additional unit test for aws-config helper when a non-default profile has been set

ginglis13 commented 1 week ago

^ latest force push serializes the two pluto unit tests such that AWS_CONFIG_FILE can be deterministically set and unset.

bottlerocket-os / bottlerocket-core-kit

Configure FIPS AWS service endpoints #218