thoughtpolice
commented
1 year ago I've just about got this working and wrinkled out most of the immediate bugs including performance, but I think I've now run into buck problems now, which may be expected.
As of a3eb8b70ce6d43a81887b382e575fd3876330f43, if you follow the above steps to start buildbarn, apply the following patch, buck kill and then try something like buck build src/hello:, it will almost work:
diff --git a/.buckconfig b/.buckconfig
--- a/.buckconfig
+++ b/.buckconfig
@@ -20,7 +20,7 @@
digest_algorithms = SHA1
[buck2_re_client]
-re_enabled = false
+re_enabled = true
engine_address = http://127.0.0.1:8980
action_cache_address = http://127.0.0.1:8980
cas_address = http://127.0.0.1:8980
diff --git a/buck/prelude/basics/download.bzl b/buck/prelude/basics/download.bzl
--- a/buck/prelude/basics/download.bzl
+++ b/buck/prelude/basics/download.bzl
@@ -18,6 +18,7 @@
[
"#!/usr/bin/env bash",
"set -xeuo pipefail",
+ "source /root/.bashrc",
"curl -Lo \"$1\" {}".format(ctx.attrs.url),
"mkdir -p \"$2\"",
"tar xf \"$1\" -C \"$2\" --no-same-owner --strip-components=1",
@@ -71,6 +72,7 @@
[
"#!/usr/bin/env bash",
"set -xeuo pipefail",
+ "source /root/.bashrc",
"curl -Lo \"$1\" {}".format(ctx.attrs.url),
"hash=$(nix hash path --type sha256 \"$1\")",
"if ! [ \"$hash\" = \"{}\" ]; then".format(ctx.attrs.hash),But alas:
austin@GANON:~/src/buck2-nix.sl$ buck build src/hello:
File changed: root//buildbarn-vm.qcow2
Action failed: prelude//toolchains:nixpkgs-overlay-rust (download_tarball https://github.com/oxalica/rust-overlay/archive/1373567ffd13719f6b7522737b010bfc514d49b4.tar.gz)
Internal error (stage: download): action_digest=c8b86439a7e637e7c7d98cf2addf516a771ed82f:93: Failed to declare in materializer: status: InvalidArgument, message: "Attempted to read a total of at least 23508808 bytes, while a maximum of 16777216 bytes is permitted", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
stdout:
stderr:
Build ID: 80d8d279-b2f7-4378-a421-2e78d6416456
RE: GRPC-SESSION-ID
Jobs completed: 4. Time elapsed: 1.1s.
BUILD FAILED
Failed to build 'prelude//toolchains:nixpkgs-overlay-rust (prelude//platform:default#632fe5438d4aecc1)'This is an ICE of sorts; the action cache in buildbarn shows it actually completed correctly and is cached properly, but it just can't be materialized as an output...
yaoddao
As of sometime late last night, there is experimental support for running a local instance of buildbarn, with the goal of being able to use
buck2's remote execution facilities to provide hermetic builds. You can go into the shell environment (i.e.cdinto this directory) and type this:This will start a QEMU virtual machine, running NixOS. You can SSH into it:
Then, once you're inside, start running buildbarn. This uses docker-compose (which is officially supported by them):
Wait a while. Now, you can visit either http://localhost:7982/ or http://localhost:7984/ to view the scheduler and blob storage web UI for buildbarn, respectively. The local GRPC API for Remote Execution is available at http://127.0.0.1:8980; you can use
grpcurl -plaintextto get at it, and this is where we point buck.Now, set the following to
truein.buckconfig, and restart buck. You can leave the API addresses as usual; they aren't used unlessre_enabledis set totrue:This will use the GRPC API and BuildBarn containers! You can see the action logs if you look at the output of the
run-buildbarncommand (which will tail the output of every docker container.) But it fails, currently:This is because the built container image doesn't have a default PATH that includes curl or anything like it — you can see this in the above output; a patch I added (but didn't commit) uses
envto dump the environment and onlyPWDandSHLVLare defined. We do need to fix this so a minimal environment works for the bash scripts. Then maybe we can get somewhere, but it will probably hit another several bumps.