fishrockz
commented
1 year ago Thank you @stagnation for writing this up
I will not that rustc will not do much more than print its version without /proc
Open stagnation opened 1 year ago
fishrockz
commented
1 year ago Thank you @stagnation for writing this up
I will not that rustc will not do much more than print its version without /proc
stagnation
commented
1 year ago I have a solution in review now. 1: https://github.com/buildbarn/bb-storage/pull/177 2: https://github.com/buildbarn/bb-remote-execution/pull/116
And a reproduction here: https://github.com/stagnation/bb-deployments/tree/feature/reproduce-bb-remote-execution-115 that uses the bare deployment to verify the patches. Currently only works on Linux.
The problem statement is simple:
programs that use /proc cannot run in chroot.
It should be easy to create a reproduction.
However, there are not publicly available programs that create a user-space input root. This is something that middleware could do, but requires a lot of code.
What options do we have?
Use the go-compiler, a bash script to call ls /proc/self, or compile a statically linked binary to do the same.
The [rules_go] toolchain builds a hermetic go compiler and sends that in the input root,
so it is a good candidate.
However it uses the system gcc to compile it.
And we get errors that gcc is not available.
An interpreted program like bash does not work either,
as we do not have the interpreter available.
Thankfully it is simple to create a statically linked go program
go_binary(
...
pure = "on",
)And we can use the eminent run_binary rule,
which does not require bash to execute a tool.
We use the compiled program directly as a source in the tool attribute.
If we were to use native_binary we get snagged up by a bash dependency
in its CopyFile action.
But with this we fail, here is the output from bb-browser:
Command^*
Arguments: ./ls-proc bazel-out/k8-fastbuild/bin/out
Environment variables: PATH=/bin:/usr/bin:/usr/local/bin
Result
Status: Code 3: Failed to run command: Failed to start process: fork
/exec /usr/bin/env: no such file or directoryThis is because Buildbarn itself wraps the process with /usr/bin/env.
Here env is used for PATH resolution before chroot.
Thankfully busybox has easy to use programs, that can help in a pinch,
especially the musl versions.
$ docker run -v .:/out busybox:musl sh -c "cp /usr/bin/env /out/env" $ file env env: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, stripped
Now we just need to put it in the input root as /usr/bin/env.
$ mkdir -p usr/bin $ mv env usr/bin/
With the "introspection" target
run_binary(
name = "introspection",
outs = ["out"],
args = ["$(location out)"],
execution_requirements = {
"no-cache": "1",
},
tool = ":ls-proc",
srcs = ["usr/bin/env"], # env is required in Buildbarn's process creation.
)With the customary docker-compose setup.
bazel build \
--remote_executor=grpc://localhost:8980 \
--remote_instance_name=fuse \
--remote_default_exec_properties OSFamily=linux \
--remote_default_exec_properties container-image="docker://ghcr.io/catthehacker/ubuntu:act-22.04@sha256:5f9c35c25db1d51a8ddaae5c0ba8d3c163c5e9a4a6cc97acd409ac7eae239448" \
@//:introspectionThis works with regular runners,
but not chroot runners.
A setback! The docker-compose setup does not build the docker images,
so we would have to create deliverables from the pull requests,
which is tedious.
Instead we use the bare deployment
where we do compile the runner from source.
So we instead build with:
bazel build \
--remote_executor=grpc://localhost:8980 \
--remote_instance_name="" \
//:introspectionAnd instead start it through bazel run:
$ mktemp -d
/tmp/tmp.sjR6aHjhni
$ bazel run --script_path launch-bare //bare
# or bazelisk if you use that, the super user typically does not have bazel on its PATH.
# or --run_under sudo
$ sudo launch-bare /tmp/tmp.sjR6aHjhniNotice that we no longer use FUSE, but for our reproduction that is okay. In production use you do want FUSE
go_repository(
name = "com_github_buildbarn_bb_remote_execution",
importpath = "github.com/buildbarn/bb-remote-execution",
- sum = "h1:BKoGfhCfn5IA4JRLMB7I4yHsM06fLvOc/zwzSxEuNrY=",
- version = "v0.0.0-20230905173453-70efb72857b0",
+ remote = "https://github.com/stagnation/bb-remote-execution",
+ vcs = "git",
+ commit = "01729791c366b6d713bf4f5e6c706cb274292539",
)
+
go_repository(
name = "com_github_buildbarn_bb_storage",
importpath = "github.com/buildbarn/bb-storage",
- sum = "h1:z9yMGmzNNjhC2KnxYGfP8bPk1/l3jpd3+rb+1YkhQg4=",
- version = "v0.0.0-20230905110346-c04246b462b6",
+ remote = "https://github.com/stagnation/bb-storage",
+ vcs = "git",
+ commit = "58bf2fed198fad8d60af26419fa0548e897165a8",
)INFO: From RunBinary out:
arch_status
attr
autogroup
auxv
cgroup
clear_refs
cmdline
comm
coredump_filter
cpu_resctrl_groups
cpuset
Target //:introspection up-to-date:
bazel-bin/out
INFO: Elapsed time: 0.306s, Critical Path: 0.03s
INFO: 2 processes: 1 internal, 1 remote.
INFO: Build completed successfully, 2 total actionsP.S: Sorry for the spam, I wish github wouldn't be quite so verbose about my pushes.
fishrockz
commented
6 months ago I had a bit of a surge of motivation to try this out recently, i normally use the docker compose in bb-deployments to use buildbarn is there a similarly easy way to try this out?
What's the recommended way to try out a patch like this if your not used to setting up buildbarn?
stagnation
commented
6 months ago Hi! Thank you, I do not have pre-built docker images, and have not rebased the patches in a while. Give me some time and I can prepare that for you, then you should be able to build the images locally and use them through docker-compose. That would be a good use-case to document too.
stagnation
commented
6 months ago I have rebased the PR now and will try to get it over the finish line soon.
Here is the instructions you need in our documentation repo. https://meroton.com/docs/improved-chroot-in-buildbarn/reproducing-the-problem/ They should help you run the bare deployment.
Problem Description / Feature Request
Using chrooted actions is great for fully hermetic input roots, but some tools rely on the
/procfilesystem to be mounted and will not work without it. The block devices from/devcan all be mounted (or created) into the input root, but full filesystems are trickier.This lists some tools that do not work without
/proc, shows a patch I used to work around it and discusses the limitations of the patch and what we need before we can merge it.Problems
In building
bb-deploymentsI found the following issues::and::
It was easy to find the root cause for the go problem::
It tries to find the go binary by looking at itself through
/proc, and then find the defaultGOROOTstandard library relative to the compiler. The proc filesystem is core to Unix/Posix, or some such standard execution environment specification and for chrooted actions to work for all tools we should have it.This is also a problem for rust, which has the best error message of the bunch https://buildteamworld.slack.com/archives/CD6HZC750/p1692391421607939
::
Proof of Concept Patch
Note, this was developed in February, and has not been rebased on recent work, https://github.com/buildbarn/bb-remote-execution/commit/1524fef4bf2044fef95210437152679e36cf00a8 which adds PATH lookup to the same file. Though the patch is orthogonal. ::
Problems with merging
Note that the absolute path is constructed here. Because the
mountsyscall does not work with relative paths. So theDirectoryabstraction in the runner, where the directory is hidden, cannot be used for these mounts.Mountat
The solution is to use the newer mount syscall api to construct a "mountat" functionality, which can mount with relative filepaths. (What the "at" suffix typically means). But when I prototyped this I could not find a way to unmount the paths.
::
This can be used in go as well, last I checked there was a PR in review for these new syscalls, but to create a wrapper for what we need is not hard. The problem is how to unmount the "proc" and "sys" relative paths after the action. According to the documentation it should work, but I could not piece it together.
TODO ++++
I have a lot more to say on this topic, and should wrap up my investigation and publish that. and should publish the scaffolding c and go code to use them, but the files are too long to paste in here.