Support Nitro Enclaves for storing ACM certificates used by Nginx/Apache

[thedevopsmachine]

What I'd like: AWS EC2 instances with Nitro support a feature called "Nitro Enclaves", which greatly enhance security when doing crypto operations, as it stores private keys in a way that they are not accessible to generic processes running on the host.

Most importantly, it has an integration with AWS Certificate Manager that allows you to deploy ACM-managed certificates onto the instances for on-host TLS termination (see https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave-refapp.html)

This is a game-changing feature for anyone who uses Nginx/Apache Httpd to terminate TLS on their hosts, both for security (since there's no private keys on the file system) and for operations (since admins no longer need to deploy the private key to the system). It would be great if Bottlerocket could support this feature, as it would greatly simplify the process of managing TLS certificates for people who use these servers.

Software Developers may want to take advantage of Nitro Enclaves as well, but I suspect that most Bottlerocket customers will primarily be interested in the ACM integration.

Any alternatives you've considered: None

[yeazelm]

Thanks for cutting this issue @thedevopsmachine! This does seem like a great thing to look into.

After looking at this a bit, it looks like the https://github.com/aws/aws-nitro-enclaves-cli (or something that does the same thing) would be required to manage the enclaves in the OS. There are some bits we might need to figure out like https://github.com/aws/aws-nitro-enclaves-cli/blob/main/bootstrap/nitro-enclaves-allocator which uses shell. Nonetheless, getting the enclave management bits working in Bottlerocket would enable users to use something like https://github.com/aws/aws-nitro-enclaves-with-k8s to leverage them as well. There is a bit of design and engineering to be done to get the nitro-cli working in Bottlerocket.

We don't have this on our roadmap right now but we'll keep this as a feature request. Thanks again for cutting this issue.

[jalaziz]

We use Bottlerocket on EKS and are looking for Nitro Enclave support specifically for the Nitro Enclaves with K8s functionality. Our use case is not TLS termination, but we'd love to see support for Nitro Enclaves in Bottlerocket.

[yeazelm]

I did a bit more investigation into this issue. This is what I found, the first problem preventing aws-nitro-enclaves-cli from being easily installed into Bottlerocket is the nitro-enclaves-allocator which is typically installed via a systemd service. This is a pretty complex bash script which will not work on Bottlerocket due to the lack of shell. We have two options to work around this problem:

• Rewrite this script in Rust (much of the rest of code is in Rust so this might be an aspirational goal for the project). Ideally this would be upstreamed into the nitro enclaves repo to enable all other consumers to use this instead of the shell script. If it isn't accepted upstream, Bottlerocket could maintain it as part of its codebase.
• Find a way to run this service as a host container or DaemonSet. The primary problem is this tooling needs access to the /dev/nitro_enclaves device as well as rw access to /sys/module/nitro_enclaves/ as well as other potential places. This means adding changes to the podspec. I got this one working in a hacky way by sidestepping the normal /sys/ path which is ro by default. I did this by adding a /hostsys path as rw and changed the code in the shell script to use that path. This only sort of worked and needs a bit more playing around to get something working. The access to the device might involve a bit more work to make sure its done correctly, I need to learn a bit more on whether providing this to all containers would be suitable or not. In theory, you might be able to fully customize the podspec to work around all these problems with privileged: true but this may not be desirable for a production environment.

Either way when working around this, the https://github.com/aws/aws-nitro-enclaves-k8s-device-plugin doesn't seem to pick this up so there is probably a bit more work to dive into this code and figure out how to get it working to enable an "out of the box" working experience in EKS.

[arnaldo2792]

I got it to work with a few hacks:

• From the admin container (yes, too much privilege), I ran /usr/bin/nitro-enclaves-allocator
• After the hugepages are configured, one must restart the kubelet otherwise it won't pick up the configurations applied (see this)

bash-4.2# /home/run.sh
Start allocating memory...
Started enclave with enclave-cid: 17, memory: 128 MiB, cpu-ids: [1, 5]
{
  "EnclaveName": "hello",
  "EnclaveID": "XXXXXXXXX",
  "ProcessID": 17,
  "EnclaveCID": 17,
  "NumberOfCPUs": 2,
  "CPUIDs": [
    1,
    5
  ],
  "MemoryMiB": 128
}
-------------------------------
Enclave ID is XXXXXXX
-------------------------------
[   1] Hello from the enclave side!

So it works in Bottlerocket :tada: ! However, the experience isn't great.

We could either allow bootstrap-containers to modify hugepages/CPU pools (there might be missing some capabilities and they already have access to all the devices in the host), so that the host is configured on boot (I don't think allowing to change the hugepages/CPU pools at runtime would safe, if there are already workloads using them). Or, we provide a sub-command like apiclient setup-enclaves <blah>, similar to the command that will be added here. With this, the experience will be similar to what EKS provides today (see this), where the enclaves are configured on boot before the kubelet runs and the workloads just work after they are deployed.

[yeazelm]

One note, you have to install file as well as the nitro enclave cli to get it working in the admin container:

amazon-linux-extras install aws-nitro-enclaves-cli file -y

Otherwise you see the error:

[root@admin]# /usr/bin/nitro-enclaves-allocator
Auto-generating the enclave CPU pool by using the CPU count...
Will try to reserve 768 MB of memory on node 0.
Configuring the huge page memory...
/usr/bin/nitro-enclaves-allocator: line 188: file: command not found
Error: Failed to find NUMA node for a CPU. This indicates an invalid SysFS configuration.