Does not build when ccache is installed

[digitalentity]

Expected Behavior

bazelisk build //... succeeds

Actual Behavior

$ bazelisk build //... 
INFO: Analyzed 60 targets (0 packages loaded, 0 targets configured).
INFO: Found 60 targets...
ERROR: /home/user/.cache/bazel/_bazel_user/03cb7a8cd9b04e5097b111baa58d2534/external/zlib/BUILD.bazel:32:11: Compiling uncompr.c [for tool] failed: (Exit 1): gcc failed: error executing command (from target @zlib//:zlib) /usr/lib/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 26 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
ccache: error: Failed to create temporary file for /home/user/.ccache/tmp/uncompr.stdout: Read-only file system
ERROR: /home/user/dev/enola/common/protobuf/BUILD.bazel:41:13 Building common/protobuf/libprotobuf.jar (6 source files) failed: (Exit 1): gcc failed: error executing command (from target @zlib//:zlib) /usr/lib/ccache/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG -ffunction-sections ... (remaining 26 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
INFO: Elapsed time: 0.243s, Critical Path: 0.05s
INFO: 9 processes: 9 internal.
FAILED: Build did NOT complete successfully

Steps to Reproduce the Problem

1. Follow the https://docs.enola.dev/dev/setup/
2. Run bazelisk build //...

Other

The --sandbox_debug is not really helpful in understanding why the sandbox is getting mounted read-only.

[vorburger]

@digitalentity thanks for your interest in building - and welcome to the party! 🥳

It does 😼 build, of course - at least on:

1. Ubuntu latest, GitHub Action default image - as proven by the green "badge" on the README and the green tick on the last commits and e.g. this build log
2. Fedora (yours truly's workstation)
3. though-shall-not-name Debian-ish-but-not-quite gOS

but what you are obviously really saying here is that it does not build for you on... where?

• Mac OS?
• Windohze WSL? (note also unrelated #147 - PITA!)
• in-Container?

Please provide as much specific further details about your exact environment where you are hitting this, ideally with steps how to reproduce this - and I'm sure we'll eventually figure this out...

Just to double check, you DID install the documented pre-requisites, and you DO indeed e.g. have a gcc available - it's hopefully not just as simple as that? That would be no fun! 🥇

/home/user/.ccache/tmp/uncompr.stdout: Read-only file system

This is kind of interesting... presumably user is edited, and that's your 👪 $HOME ?

But why is your ~/.ccache/tmp/ on a RO FS? This smells funny...

[digitalentity]

Hey hey!

I spent the evening pulling my hair out trying to figure it out - to no avail :facepalm:

Some details about the system:

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.6 LTS"

$ uname -a
Linux homestation 5.15.0-56-generic #62~20.04.1-Ubuntu SMP Tue Nov 22 21:24:20 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ gcc --version
gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ go version
go version go1.13.8 linux/amd64

$ python3 --version
Python 3.8.10

$ bazelisk --version
bazel 6.1.2

Trying to build in the linux environment, without containers. There are really no specific steps to repro - run bazelisk build and see the failure happening. Obviously all of the documented pre-requisites are installed.

Fun part is that ~/.ccache/tmp/ is definitely not on the RO FS, I even tried making it world writable - to no avial. I guess Bazel's linux-sandbox is doing something funky, but I have no idea what or how to debug.

[vorburger]

https://github.com/bazelbuild/bazel/issues/12124 seems relevant here...

I just realized that I do not actually use ccache (as in, I don't even have it installed), so the current working theory here, and tentative new title (until proven otherwise) is that this is related to that. I don't suppose sudo apt-get remove ccache is an acceptable option for you? 😸 But even if not, would you be willing to try it at least temporarily (and re-install immediately after) - just to confirm that's it?

If not, the aforementioned bug has number of other tips; --spawn_strategy=standalone in particular should help (from what I understand about Bazel internals). It's a shame though - you would loose half of what makes Bazel fun - but perhaps for now, to unblock...

[vorburger]

https://stackoverflow.com/questions/52370202/how-to-get-bazel-ccache-and-sandboxing-to-work-together-ccache-read-only-file is also about this, and seems to suggest to add --sandbox_writable_path=$HOME/.ccacheto your ~/.bazelrc - wanna try that and see if that helps?

[digitalentity]

I'm trying to do a clean setup via docker run -it ubuntu and installing prerequisites fails here:

# go install github.com/bazelbuild/bazelisk@latest
can't load package: package github.com/bazelbuild/bazelisk@latest: cannot use path@version syntax in GOPATH mode

What fixed it is falling back to go get:

# go get github.com/bazelbuild/bazelisk
# export PATH=$PATH:$(go env GOPATH)/bin
# bazelisk --version
2023/05/15 22:11:21 Downloading https://releases.bazel.build/6.2.0/release/bazel-6.2.0-linux-x86_64...
bazel 6.2.0
# bazelisk build //...
INFO: Analyzed 60 targets (1 packages loaded, 9 targets configured).
INFO: Found 60 targets...
INFO: Elapsed time: 4.574s, Critical Path: 0.00s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action

So we know that there is no problem with the toolchain and it works, at least within a docker container. Now the challenge is to get it running without "metal boxes".

Tried $ sudo apt remove ccache - didn't work, now Bazel can't find gcc, though apparently gcc --version still works...

Adding --sandbox_writable_path=$HOME/.ccache didn't help either, still complains about RO filesystem and fails :facepalm:

--spawn_strategy=standalone is what finally made it build. A single-threaded build is pain though, even make can do better :stuck_out_tongue:

[digitalentity]

Let's keep this open, I'll see if I can make it work with ccache.