search

bottlerocket-os / bottlerocket

An operating system designed for hosting containers
https://bottlerocket.dev
Other
8.78k stars 519 forks source link

Ephemeral Storage commands have a race condition when used with bootstrap commands #4181

Closed yeazelm closed 2 months ago

yeazelm commented 2 months ago

Image I'm using: Proposed images for k8s and ecs built from develop on the latest kit v2.4.0.

What I expected to happen: Using the bootstrap commands to invoke the ephemeral storage commands, I expect the local storage to be added to an array and mounted for the paths called out in the code: k8s

[settings.bootstrap-commands.000-test-bootstrap-commands]
commands = [[ "apiclient", "ephemeral-storage", "init"],
["apiclient", "ephemeral-storage", "bind", "--dirs", "/var/lib/containerd", "/var/lib/kubelet", "/var/log/pods"]]
essential = false
mode = "once"

ECS

[settings.bootstrap-commands.000-test-bootstrap-commands]
commands = [[ "apiclient", "ephemeral-storage", "init"],
["apiclient", "ephemeral-storage", "bind", "--dirs", "/var/lib/containerd", "/var/lib/docker", "/var/log/ecs"]]
essential = false
mode = "once"

I expect to see the bootstrap commands to succeed:

# journalctl -D /.bottlerocket/rootfs/var/log/journal/ -u bootstrap-commands
localhost systemd[1]: Starting Bootstrap Commands...
localhost bootstrap-commands[6281]: 22:42:47 [INFO] Processing bootstrap command '000-test-bootstrap-commands' ...
internal bootstrap-commands[6281]: 22:42:48 [INFO] Turning off bootstrap command '000-test-bootstrap-commands'
internal bootstrap-commands[6281]: 22:42:48 [INFO] Successfully ran bootstrap command '000-test-bootstrap-commands'
internal systemd[1]: Finished Bootstrap Commands.

What actually happened: I see failures with the following issue:

localhost bootstrap-commands[6007]: 22:41:16 [INFO] Processing bootstrap command '000-test-bootstrap-commands' ...
internal bootstrap-commands[6226]: Failed to initialize ephemeral storage: Failed POST request to '/actions/ephemeral-storage/bind': Status 500 whenPOSTing /actions/ephemeral-storage/bind: Unable to bind ephemeral storage: Failed to canonicalize path, No such file or directory (os error 2)
internal bootstrap-commands[6007]: 22:41:17 [INFO] Turning off bootstrap command '000-test-bootstrap-commands'
internal bootstrap-commands[6007]: 22:41:17 [INFO] Successfully ran bootstrap command '000-test-bootstrap-commands'
internal systemd[1]: Finished Bootstrap Commands.

This seems to be an issue with https://github.com/bottlerocket-os/bottlerocket-core-kit/blob/develop/sources/api/apiserver/src/server/ephemeral_storage.rs#L217 where the path isn't created yet for the /dev/disk/by-label/.

How to reproduce the problem: Build the latest commit, and pass the above user-data to an instance that has Instance Storage attached.