rules_go sandboxing issue

[Globegitter]

I just started to test out the new rust sandboxfs implementation and upon building one of our go applications ran into an issue. First this was an issue in sandboxfs itself, but after reporting the issue there, it seems the issue is within rules_go itself, see https://github.com/bazelbuild/sandboxfs/issues/67 for the actual bug report as well as the investigation on the issue.

[jmmv]

Just to summarize the problem reported in the sandboxfs tracker, the Go rules are putting external/go_sdk/bin/go into the sandbox but not the rest of the Go toolchain. With the standard symlink-based sandboxing, this works because external/go_sdk/bin/go is a symlink to the original directory and Go can discover the rest of the files (after escaping the sandbox). With the stricter sandboxfs-based sandboxing, Go doesn't know how to find its extra files (because external/go_sdk/bin/go is not a symlink anymore). Maybe the whole toolchain should be supplied to the action as an input?

[jayconrod]

Hmm, all of the necessary SDK files are registered as inputs (everything except the pre-compiled standard library for the host, which shouldn't be needed for cross-compilation). This works with remote execution, so I wouldn't expect missing inputs would be a problem.

Is it possible GoStdLib is triggering a sandboxfs bug here? The GoStdLib action basically copies the src, pkg/tool, and pkg/include directories to a new location, then runs go install std runtime/cgo. The output is a tree artifact (which has triggered bugs in other implementations).

I tried to reproduce this, but it looks like setting up sandboxfs on macOS will take more time than I have today (it requires Rust, Brew, pkg-config, osxfuse at least).

[jmmv]

I don't think this is a sandboxfs bug. This is what sandboxfs is asked to map:

/3084
/3084/bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/info
/3084/bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/info.runfiles/io_bazel_rules_go/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/info
/3084/bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/info.runfiles/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/info
/3084/external/go_sdk/bin/go

and Bazel doesn't do anything particularly special here: it just passes the list of inputs for an action pretty much verbatim to sandboxfs. Note that the only thing under external/go_sdk is bin/go.

I've also tried to pass --sandbox_debug when not using sandboxfs (thus using Bazel's typical symlink sandboxing) and the result is the same: the execroot for the sandbox only contains external/go_sdk/bin/go, but this is a symlink to the real SDK... so Go escapes the sandbox as it follows that symlink.

Also, the only two actions that fail in the whole build are GoInfo (whatever that is). Maybe that single rule is the one with a bug?

(I should have a release for sandboxfs out soon, which will come with prebuilt packages and thus simplify installation. But this bug concerns me slightly so I might hold off before doing that until we know what's happening.)

[jmmv]

I don't know if this is the exact solution, but this fixes the problem:

diff --git a/go/private/rules/info.bzl b/go/private/rules/info.bzl
index a63426a..66c5468 100644
--- a/go/private/rules/info.bzl
+++ b/go/private/rules/info.bzl
@@ -27,7 +27,7 @@ def _go_info_impl(ctx):
     args = go.builder_args(go)
     args.add("-out", report)
     go.actions.run(
-        inputs = [go.go],
+        inputs = [go.go] + go.sdk_files,
         outputs = [report],
         mnemonic = "GoInfo",
         executable = ctx.executable._go_info,

[Globegitter]

@jayconrod Just fyi the issue can also be reproduced on Linux, imo the manual compilation is slightly "simpler" there.

[jayconrod]

Oh is go_info the only thing that's broken? It sounded like it was go_stdlib.

I've made the change @jmmv proposed in #1917. Could one of you please take a look and verify?

[jmmv]

Hmm... that's right, the original message from @Globegitter hinted at a different issue. The patch I proposed works for me, but maybe there is some other different issue when running on Linux instead of macOS. Would be good if @Globegitter confirms.

[Globegitter]

I just tried https://github.com/bazelbuild/rules_go/pull/1917 and I can build that just fine. I will also test this now on our go appplications to see if that also fixed building those.

[Globegitter]

Still running into issues in building the go app I just tested, getting the following error:

(08:28:43) INFO: Mounting sandboxfs instance on /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs
(08:28:47) ERROR: /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/external/io_bazel_rules_go/BUILD.bazel:9:1: GoStdlib external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/pkg failed (Exit 1) stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
can't load package: package os/signal/internal/pty: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/5/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/os/signal/internal/pty
can't load package: package runtime/msan: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/5/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/runtime/msan
can't load package: package syscall/js: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/5/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/syscall/js
can't load package: package vendor/golang_org/x/net/lif: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/5/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/vendor/golang_org/x/net/lif
can't load package: package vendor/golang_org/x/net/route: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/5/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/vendor/golang_org/x/net/route
GoStdlib: error running subcommand: exit status 1

I will see if I can get a reproduction repo online.

Edit: Actually that is also failing without sandboxfs but just with a very different error message:

(08:35:28) ERROR: /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/external/com_github_rs_cors/BUILD.bazel:3:1: GoCompile external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a failed (Exit 1) compile failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CGO_ENABLED=0 \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_rs_cors/cors.go -src external/com_github_rs_cors/utils.go -o bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -nogo bazel-out/host/bin/linux_amd64_stripped/nogo -x bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.x -p github.com/rs/cors -- -trimpath .)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: compile failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CGO_ENABLED=0 \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_rs_cors/cors.go -src external/com_github_rs_cors/utils.go -o bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -nogo bazel-out/host/bin/linux_amd64_stripped/nogo -x bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.x -p github.com/rs/cors -- -trimpath .)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
GoCompile: nogo: internal error: gob encoding of analysis fact printfWrapper failed: gob: type not registered for interface: printf.isWrapper; please report a bug against fact *printf.isWrapper in package "golang.org/x/tools/go/analysis/passes/printf"
panic: internal error: gob encoding of analysis fact printfWrapper failed: gob: type not registered for interface: printf.isWrapper; please report a bug against fact *printf.isWrapper in package "golang.org/x/tools/go/analysis/passes/printf"

goroutine 1 [running]:
log.Panicf(0x6aac3e, 0x6e, 0xc000c11c78, 0x4, 0x4)
    GOROOT/src/log/log.go:333 +0xda
golang.org/x/tools/go/analysis/internal/facts.(*Set).Encode(0xc000b206c0, 0x5, 0x8, 0xc0001198b0)
    external/org_golang_x_tools/go/analysis/internal/facts/facts.go:266 +0x7ad
main.checkPackage(0x8807c0, 0x5, 0x5, 0x7ffebafe9be6, 0x12, 0xc000081440, 0xc000081470, 0xc0000814d0, 0xc000092130, 0x2, ...)
    external/io_bazel_rules_go/go/tools/builders/nogo_main.go:177 +0x308
main.run(0xc000092010, 0x14, 0x15, 0x0, 0x0)
    external/io_bazel_rules_go/go/tools/builders/nogo_main.go:80 +0x3cc
main.main()
    external/io_bazel_rules_go/go/tools/builders/nogo_main.go:54 +0x8d
Target //app/cmd/app:app.binary failed to build
(08:35:28) ERROR: /home/markus/Projects/github.com/ecosia/core/app/cmd/app/BUILD.bazel:27:1 GoCompile external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a failed (Exit 1) compile failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CGO_ENABLED=0 \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_rs_cors/cors.go -src external/com_github_rs_cors/utils.go -o bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -nogo bazel-out/host/bin/linux_amd64_stripped/nogo -x bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.x -p github.com/rs/cors -- -trimpath .)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: compile failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CGO_ENABLED=0 \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib% \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/compile -sdk external/go_sdk -installsuffix linux_amd64 -src external/com_github_rs_cors/cors.go -src external/com_github_rs_cors/utils.go -o bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.a -package_list bazel-out/host/bin/external/go_sdk/packages.txt -nogo bazel-out/host/bin/linux_amd64_stripped/nogo -x bazel-out/k8-fastbuild/bin/external/com_github_rs_cors/linux_amd64_pure_stripped/go_default_library%/github.com/rs/cors.x -p github.com/rs/cors -- -trimpath .)
Execution platform: @bazel_tools//platforms:host_platform

So maybe the failure is not due to sandboxfs as nogo disabled I can compile the app with and without sandboxfs without any problems.

[Globegitter]

I was just testing another of our go apps (having nogo deactivated again), which works fine without sandboxfs but with sanboxfs I am getting the following error:

(08:41:57) ERROR: /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/external/io_bazel_rules_go/BUILD.bazel:9:1: GoStdlib external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/pkg failed (Exit 1) stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox
can't load package: package os/signal/internal/pty: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/406/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/src/os/signal/internal/pty
can't load package: package runtime/msan: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/406/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/src/runtime/msan
can't load package: package syscall/js: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/406/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/src/syscall/js
can't load package: package vendor/golang_org/x/net/lif: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/406/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/src/vendor/golang_org/x/net/lif
can't load package: package vendor/golang_org/x/net/route: build constraints exclude all Go files in /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/sandbox/sandboxfs/406/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/src/vendor/golang_org/x/net/route
GoStdlib: error running subcommand: exit status 1
Target //other-app/server:other-app failed to build
(08:41:57) ERROR: /home/markus/Projects/github.com/ecosia/core/other-app/server/BUILD.bazel:32:1 GoStdlib external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib%/pkg failed (Exit 1) stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

Use --sandbox_debug to see verbose messages from the sandbox: stdlib failed: error executing command 
  (cd /home/markus/.cache/bazel/_bazel_markus/c83dd5dc66625baf386a59b750f50a10/execroot/__main__ && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=0 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/stdlib -sdk external/go_sdk -installsuffix linux_amd64 -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_static_pure_stripped/stdlib% -filter_buildid bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/linux_amd64_stripped/filter_buildid)
Execution platform: @bazel_tools//platforms:host_platform

This is a project using protobufs/grpc (the other one is not), but not sure if that has anything to do with the failure. Again will see if I can put up a reproduction repo.

[jmmv]

@Globegitter If you could come up with a reproduction case "soon", it'd be awesome. I want to cut a sandboxfs release but this issue makes me nervous and would like to figure out what's going on beforehand.

I've just tried building a Go project that uses protobufs and couldn't reproduce though...

[Globegitter]

@jmmv @jayconrod sorry for the delay here a repro repository: https://github.com/Globegitter/sandboxf-go-repro

just run: bazel build --verbose_failures //app --experimental_use_sandboxfs

Setting pure="on" on the binary seems to cause the issues. We also use static="on" but that seems to have no effect.

[Globegitter]

@jayconrod I can still reproduce the unrelated nogo error reported above on the latest commit on master on one of our apps (but have not been able to reproduce it yet with a minimal example) - should I pull this out into a separate issue?

[jmmv]

OK, I can reproduce the problems with the given test case and command line, but only on Linux. On macOS, the problems don't appear.

The contents of the sandbox look reasonable to me: external/go_sdk is populated with what I'd expect from a GOROOT.

@jayconrod So the question is, what does "build constraints exclude all Go files in" mean? Where are these build constraints defined? May it be that these are defined in terms of paths outside of the sandbox and, because we don't use symlinks anymore, we confuse the tool?

[jayconrod]

"build constraints exclude all Go files" would be emitted by go install when it encounters a package without any .go files that are buildable for the target architecture. That should not be the case for any of these packages, so I think it means the source files are not visible to go install for some reason. We may be depending on symlinks somewhere, but it needs investigation.

[jmmv]

I see. So, for example, take this error:

can't load package: package os/signal/internal/pty: build constraints exclude all Go files in /usr/local/google/home/jmmv/.cache/bazel/_bazel_jmmv/c4483b789ce40e53e164baabc09a8e13/sandbox/sandboxfs/0/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/os/signal/internal/pty

If I check the content of that directory after the action failed:

ls -l /usr/local/google/home/jmmv/.cache/bazel/_bazel_jmmv/c4483b789ce40e53e164baabc09a8e13/sandbox/sandboxfs/0/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_pure_stripped/stdlib%/src/os/signal/internal/pty
total 0K
-rw-r--r-- 1 jmmv primarygroup 1387 Jan 29 13:41 pty.go

and the file is readable. Should I be looking for something else that might not be there?

Something else I tried is running go from within the sandbox directory to see if the GOROOT is valid:

GOROOT=external/go_sdk ./external/go_sdk/bin/go run /tmp/test.go
go build internal/race: external/go_sdk/pkg/tool/linux_amd64/compile: fork/exec external/go_sdk/pkg/tool/linux_amd64/compile: no such file or directory
[... many more errors ...]

but specifying the GOROOT as absolute works fine:

GOROOT=$(pwd)/external/go_sdk ./external/go_sdk/bin/go run /tmp/test.go
hello

so I think this is OK too.

[jayconrod]

I was able to reproduce this. I spent some time debugging, but I'm afraid I wasn't able to get very far.

To recap, this is basically what the action is doing (not literally this, but something similar):

#!/bin/bash

out=$(mktemp -d)   # actually given by Bazel
echo writing to $out >&2
goroot=$(go env GOROOT)
mkdir "$out/pkg"
cp -r "$goroot/src" "$out/src"
cp -r "$goroot/pkg/tool" "$out/pkg/tool"
cp -r "$goroot/pkg/include" "$out/pkg/include"
cd "$out"

CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64 \
GOROOT=$out \
GOCACHE=$out/.cache \
go install std

I confirmed that the files in src, pkg/tool, and pkg/include are present, and they're real files, not symbolic links.

The packages that are failing to build don't seem like they're intended to be built with CGO_ENABLED=0. For example, os/signal/internal/pty has one source file that requires cgo. It's only imported by tests, so we don't really need it. I'd normally expect go install std to exclude these packages, since they're not buildable in this configuration, but it's trying to build them anyway.

I wonder if there's some I/O error reading the source files that the list process is running into but the compile process is not? I'm not sure if the Go command would report errors when it's just listing packages to build.

[jmmv]

Were you able to reproduce this using the shell code you posted above, with $out being a directory under sandboxfs? I cannot trigger a problem this way.

[jayconrod]

No, I couldn't reproduce it using the shell code. I just meant to provide that as an explanation in case there was anything in there that obviously wouldn't work.

[jmmv]

Looking into the sandbox directory for the real failing action:

.../sandboxfs/8$ find . -type l
./external/go_sdk/ROOT
.../sandboxfs/8$ ls -l external/go_sdk/ROOT
lrwxrwxrwx 1 jmmv primarygroup 106 Feb  4 13:48 external/go_sdk/ROOT -> /usr/local/google/home/jmmv/.cache/bazel/_bazel_jmmv/c4483b789ce40e53e164baabc09a8e13/external/go_sdk/ROOT

Is that ROOT symlink, pointing outside of the sandbox, OK?

[jayconrod]

ROOT should be fine. It's just a file that marks the root directory. It shouldn't matter if it's a symbolic link or a real file.

[arjantop]

Has this been fixed? There was #1917 merged but this ticket was not closed.

Testing sandboxfs, not sure if it is the same issue but similar output:

INFO: Unmounting sandboxfs instance left behind on /cache/bazel/output_base/sandbox/sandboxfs by a previous command
INFO: Mounting sandboxfs instance on /cache/bazel/output_base/sandbox/sandboxfs
ERROR: /cache/bazel/output_base/external/io_bazel_rules_go/BUILD.bazel:8:1: GoStdlib external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/pkg failed (Exit 1) linux-sandbox failed: error executing command
  (cd /cache/bazel/output_base/sandbox/sandboxfs/292 && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
    TMPDIR=/tmp \
  /cache/bazel/user_root/install/8eab0b6f2b5ac882f3a1a7c2494aade6/linux-sandbox -t 15 -w /cache/bazel/output_base/sandbox/linux-sandbox/293/execroot/__main__ -w /tmp -w /dev/shm -D -- bazel-out/host/bin/external/go_sdk/builder stdlib -sdk external/go_sdk -installsuffix linux_amd64_race -tags race -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib% -race)
src/main/tools/linux-sandbox.cc:154: linux-sandbox-pid1 has PID 25383
src/main/tools/linux-sandbox-pid1.cc:175: working dir: /cache/bazel/output_base/sandbox/sandboxfs/292
src/main/tools/linux-sandbox-pid1.cc:194: writable: /cache/bazel/output_base/sandbox/linux-sandbox/293/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:194: writable: /tmp
src/main/tools/linux-sandbox-pid1.cc:194: writable: /dev/shm
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /proc
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/pts
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/mqueue
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/termination-log
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /dev/fuse
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /dev/shm
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/systemd
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/cpuset
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/blkio
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/cpu,cpuacct
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/hugetlb
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/devices
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/memory
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/net_cls,net_prio
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/freezer
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/pids
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /sys/fs/cgroup/perf_event
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /app
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /app/build
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /az
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /app-var
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /cache
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /cache/bazel/output_base/sandbox/sandboxfs
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /tmp
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /etc/hosts
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /etc/resolv.conf
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /etc/hostname
src/main/tools/linux-sandbox-pid1.cc:265: remount ro: /run/secrets/kubernetes.io/serviceaccount
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /cache/bazel/output_base/sandbox/sandboxfs/292
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /cache/bazel/output_base/sandbox/linux-sandbox/293/execroot/__main__
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /tmp
src/main/tools/linux-sandbox-pid1.cc:265: remount rw: /dev/shm
src/main/tools/process-tools.cc:118: sigaction(32, &sa, nullptr) failed
src/main/tools/process-tools.cc:118: sigaction(33, &sa, nullptr) failed
can't load package: package internal/syscall/windows: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/internal/syscall/windows
can't load package: package internal/syscall/windows/registry: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/internal/syscall/windows/registry
can't load package: package internal/syscall/windows/sysdll: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/internal/syscall/windows/sysdll
can't load package: package runtime/msan: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/runtime/msan
can't load package: package syscall/js: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/syscall/js
can't load package: package vendor/golang.org/x/net/lif: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/vendor/golang.org/x/net/lif
can't load package: package vendor/golang.org/x/net/route: build constraints exclude all Go files in /cache/bazel/output_base/sandbox/sandboxfs/292/bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/src/vendor/golang.org/x/net/route
stdlib: error running subcommand: exit status 1
src/main/tools/linux-sandbox-pid1.cc:437: waitpid returned 2
src/main/tools/linux-sandbox-pid1.cc:457: child exited with code 1
src/main/tools/linux-sandbox.cc:204: child exited normally with exitcode 1
ERROR: /app/shared/golang/blobs/BUILD.bazel:30:1 GoStdlib external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib%/pkg failed (Exit 1) linux-sandbox failed: error executing command
  (cd /cache/bazel/output_base/sandbox/sandboxfs/292 && \
  exec env - \
    CC=/usr/bin/gcc \
    CGO_CFLAGS='-U_FORTIFY_SOURCE -fstack-protector -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -fno-canonical-system-headers -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted"' \
    CGO_ENABLED=1 \
    CGO_LDFLAGS='-fuse-ld=gold -Wl,-no-as-needed -Wl,-z,relro,-z,now -B/usr/bin -pass-exit-codes -lstdc++ -lm' \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=external/go_sdk \
    GOROOT_FINAL=GOROOT \
    PATH=/usr/bin:/bin \
    TMPDIR=/tmp \
  /cache/bazel/user_root/install/8eab0b6f2b5ac882f3a1a7c2494aade6/linux-sandbox -t 15 -w /cache/bazel/output_base/sandbox/linux-sandbox/293/execroot/__main__ -w /tmp -w /dev/shm -D -- bazel-out/host/bin/external/go_sdk/builder stdlib -sdk external/go_sdk -installsuffix linux_amd64_race -tags race -out bazel-out/k8-fastbuild/bin/external/io_bazel_rules_go/linux_amd64_race_stripped/stdlib% -race)
INFO: Leaving sandboxfs mounted because of --sandbox_debug

[jayconrod]

@arjantop No, this hasn't been fixed. Not sure if #1917 was actually related to this.