Ways to install customized security setup

[congzhegao]

Image I'm using:ami-06d450b5cbf46331d (which is the latest version for k8s 1.25)

What I expected to happen: It should come with latest version or at least newer version of glibc

What actually happened:glibc version is 2.26 which is pretty old

How to reproduce the problem: I was just following the quickstart guide to create a eks cluster with eksctl

[stmcginnis]

Hi @congzhegao - Bottlerocket actually has glibc 2.38 with several backported patches for various CVEs and other important updates that have been identified. The glibc 2.39 release isn't planned until early next year.

Are you actually looking at the glibc in the control or admin containers? Those container images are based on Amazon Linux 2, which does still use an older glibc. The applications running in those containers are generally pretty limited. If you need to run something there that needs a newer version of glibc you could provide a custom host container.

[congzhegao]

@stmcginnis Thank you so much for the quick response! Is there a way to attach customized user data (EC2) to bottlerocket?

This is the default one if I'm following the quickstart guide

[settings]
  motd = "Hello from eksctl!"

  [settings.host-containers]

    [settings.host-containers.admin]

      # Bottlerocket Admin Container
      enabled = true

  [settings.kubernetes]

    # Kubernetes Control Plane API Endpoint
    api-server = "https://abcd.efg.us-west-2.eks.amazonaws.com"

    # Kubernetes Cluster CA Certificate
    cluster-certificate = "some certs"

    # Kubernetes Cluster Name
    cluster-name = "bottlerocket"

    [settings.kubernetes.node-labels]
      "alpha.eksctl.io/cluster-name" = "bottlerocket"
      "alpha.eksctl.io/nodegroup-name" = "ng-bottlerocket"

And I'd like to run a bash script to install ssm agent and some other security setup (and that requires root permission). I'm currently getting the following error when trying to run some script like:

#!/bin/bash
sudo yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm
aws s3 cp s3://blahblah
sudo install.sh

But I got

Starting Bottlerocket userdata configuration system...
[    3.974138] early-boot-config[1091]: Provider error: Unable to serialize settings from instance user data: Error parsing TOML user data: expected an equals, found an identifier at line 2 column 5
[FAILED] Failed to start Bottlerocket userdata configuration system.
See 'systemctl status early-boot-config.service' for details.
[DEPEND] Dependency failed for Bottlerocket initial configuration complete.
[DEPEND] Dependency failed for Isolates configured.target.
[DEPEND] Dependency failed for User-specified setting generators.
[DEPEND] Dependency failed for Applies settings to create config files.
[DEPEND] Dependency failed for Sets the hostname.
[DEPEND] Dependency failed for Send signal to CloudFormation Stack.

[stmcginnis]

And I'd like to run a bash script to install ssm agent and some other security setup

If you create a custom control container you can install whatever version of the SSM agent you would like. The current version included in the default control container is 3.2.1478.

For "other security setup", I guess that would depend what you mean by that. You should be able to control most things with the user data settings passed in, but if you need to do anything beyond that you may need to resort to also using a bootstrap container that can make additional modification.

If the existing settings don't cover what you are trying to do, maybe if you could describe your use case in a little more detail I might be able to suggest some workarounds.

[congzhegao]

Thank you! It sounds like we might need to use bootstrap container for this. I'm afraid I can't reveal too much on details of security setup, but it's mainly download a zipped package from s3 (or mount) and unzip it and run the install (bash) script with sudo.

From the doc, what I can do it to add this to the settings

[settings.bootstrap-containers.bootstrap]
source = "MY-CONTAINER-URI"
mode = "once"
essential = true

And then when the ec2 instance is started, it will run this container during bootstrap and whatever is installed will stay in the main (host) container at /mnt. I guess I can download the package in bootstrap container and install it in host container? btw if that's the case how can I pass the install command to host container so that it will run the install cmds?

[stmcginnis]

You should be able to run whatever you need just in the bootstrap container. But it sounds like maybe you are trying to install binaries onto the Bottlerocket host partition itself? That will not be possible as one of the security benefits of Bottlerocket is its read-only root filesystem. There are some writable areas, but nothing that will likely meet your needs if you are trying to add a binary or service to run.

If that is the case, then you would need to build your own variant with those binaries baked in. That would mean taking on responsibility for future updates though, so you should really consider if you need that. It could be better to figure out how to add your functionality into a container that can run on Bottlerocket, rather than trying to modify Bottlerocket itself.

[congzhegao]

Thank you for the suggestion! One more dumb question - If I build my own version of bottlerocket, do I just need to add one more package under /packages or is there an easier way to achieve this like adding extra lines somewhere in makefile?

[stmcginnis]

That's actually a good question! :)

Typically to add new software you would add a new package. It may be possible to pull a few more files in to one of the existing packages, but I think for long term maintainability it would probably be cleaner to keep it in its own separate package definition.

Here are a couple recent examples of adding new packages:

https://github.com/bottlerocket-os/bottlerocket/pull/3198 https://github.com/bottlerocket-os/bottlerocket/pull/2896

[congzhegao]

This is helpful, thank you!