mgsharm
commented
2 months ago @fheinecke Thank you for bringing this to our attention. We will investigate and provide an update soon.
Open fheinecke opened 2 months ago
mgsharm
commented
2 months ago @fheinecke Thank you for bringing this to our attention. We will investigate and provide an update soon.
rpkelly
commented
2 months ago I spent some time looking at this today, and unfortunately I have not yet solved it. I was able to confirm that the same issue occurs when using a fedora container, so the error is unlikely to be related to the environment.
jmt-lab
commented
2 months ago Hello, Unfortunately you are running into a unique case of docker handoff triggered by how we do docker builds in twoliter for bottlerocket. So during build we mount the bottlerocket sources and build directories inside of a docker build context in which we perform all the builds using our build tools and the bottlerocket-sdk.
However in your case you are running the build from inside a docker container, with the docker daemon causing an interesting situation. For the purposes of this explanation lets define some terms:
HOST - Where the docker daemon is BUILD - Where twoliter is orchestrating builds (has bottlerocket-sdk as a base) using buildkit (docker buildx) CONT - Where you are running the above commands
When you run cargo make inside CONT the build flow is then trying to take folders inside of CONT's VOLUME and mount it as a volume inside of BUILD which is running on HOST. So essentially what is happening is that the build system is trying to tell another parallel docker container to mount a subsection of another docker container's volume that the daemon just fails out on. There is a possibility that this could be made to work however in the future depending on how complex handling this would be.
You should however be able to build bottlerocket using Docker-in-Docker setup. I was able to get a build to work using a similar setup to the official docker:dind image (though mine is based on fedora for ease of dependency installations, but should also work using ubuntu)
None-the-less i have created two issues in our build tool Twoliter to potentially look into improving this situation.
fheinecke
commented
2 months ago Thanks for the investigation @jmt-lab. To clarify, when you say "Docker-in-Docker", you mean a separate Docker daemon/socket/etc within CONT, as opposed to the Docker daemon/socket/etc within HOST, but accessed from CONT, correct?
If possible it would be convenient for us to be able to build Bottlerocket from within a container. This would help us in a couple of areas:
This is definitely something that we can work around with DinD - it would just be nice to have support for building with a shared docker socket.
jmt-lab
commented
2 months ago Yes when i mention Docker-in-Docker I do mean using a docker daemon within CONT in a privileged container. Currently this works. I do agree that ideally your workflow should work and I'm doing some investigations today with volume mounts to see if there is a workaround that will work in the meantime before we can properly investigate a fix.
jmt-lab
commented
2 months ago I have made a breakthrough and figured out a way forward for this workflow and It is not as bad as I thought it would be.
First ubuntu does not by default install buildx when you install docker.io with the --no-install-recommends, we required buildx internally and i think this is part of the problem so you need to add docker-buildx to your apt install (embarassing to me that i didn't catch that earlier XD).
Second unfortunately the bottlerocket git clone needs to pre-exist on the host before you perform the build. This is because in order to improve the build time we share file descriptors into the build context. If these file descriptors are not on the host before docker runs the build context does not seem to be aware of them.
Secondly my 'trick' is to have a shared volume mount between the host and the container for the work so buildx can mount things correctly. This though requires identical paths between HOST and CONT. In addition you need to share the /tmp folder with the docker container as that is also used by the build. Below is a script i ran to successfully build a bottlerocket variant inside docker sharing a docker daemon:
#!/usr/bin/env bash
git clone https://github.com/bottlerocket-os/bottlerocket.git
cd bottlerocket
BOTTLEROCKET_PATH="$(pwd)"
cat <<EOF >> build.sh
#!/usr/bin/env bash
apt update
apt install -y --no-install-recommends docker.io docker-buildx
docker image ls
apt install -y build-essential openssl libssl-dev pkg-config liblz4-tool
apt install -y rustup git curl
rustup install stable || true
cargo install cargo-make
cd "${BOTTLEROCKET_PATH}"
cargo make
EOF
docker run --rm -it -v "/tmp:/tmp" -v "/var/run/docker.sock:/var/run/docker.sock" -v "${BOTTLEROCKET_PATH}:${BOTTLEROCKET_PATH}" ubuntu sh -c "cd ${BOTTLEROCKET_PATH} && chmod +x ./build.sh && ./build.sh"
Hi folks,
I'm working on building out some Bottlerocket OS variants (this is related to Kata Container work). As a first step, I'm attempting to build the project as-is, with zero changes, exactly as the docs state.
I've deployed a fresh c7a.8xlarge instance in one of our dev AWS accounts and I'm using it for initial development. I'm able to build the project (thanks for your help Matt!) just fine when running directly on this EC2 instance/VM, but not from within a container image. Also as a side note, the build instructions are missing
curlas a dependency which is required by the twoliter setup script.Here is a (long) two-line command to reproduce the issue, which should be sufficient to show that this isn't due to something cached or pre-existing state of some form:
This is pretty difficult to read, so here's what the build shell command broken down in a less copy-pastable format:
This fails with:
This exits with exit code
105.Platform I'm building on: Ubuntu 24.04 (host and container image)
What I expected to happen:
cargo maketo succeed when ran inside a containerWhat actually happened: It failed with exit code 105 (see logs above)
How to reproduce the problem: See above