Non-reproducible builds with templated files

[tommyknows]

What version of rules_go are you using?

v0.26.0

What version of gazelle are you using?

V0.23.0

What version of Bazel are you using?

4.0.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Linux / MacOS, AMD64.

What did you do?

At work, we are templating Go files with information from other parts of the build. For example, go_image (+ container_push) produce a <target>.digest file, containing the image digests. These digests are then added to .go files, which are further used within the build process.

All images are built with goos = "linux" and goarch = "amd64".

So far, we always copied those templated files back into the workspace with a bazel run command. However, as I need those templated files now as part of the build-process already, I do not want to need multiple bazel runs to get to the end state.

To work around this, we depend on the templated, but not yet copied into the workspace, file within go_library.

What did you expect to see?

To get the same final image digest, no matter on which platform the build is being run.

What did you see instead?

Non-reproducible builds with different libraries being built on MacOS and Linux. I think the cause of this issue is the path of the templated file, which seems to differ on platforms. MacOS:

Writing to output file bazel-out/darwin-fastbuild-ST-5fa8105c1248/bin/hello.go

Linux:

Writing to output file bazel-out/k8-fastbuild-ST-5fa8105c1248/bin/hello.go

This seems to build different libraries: MacOS:

$ ls -la bazel-out/darwin-fastbuild-ST-5fa8105c1248/bin/
-r-xr-xr-x  1 ramon  wheel  7852 Mar 12 11:27 mybin.a
-r-xr-xr-x  1 ramon  wheel   160 Mar 12 11:27 mybin.x

Linux:

$ ls -la bazel-out/k8-fastbuild-ST-5fa8105c1248/bin/
-r-xr-xr-x 1 root root 7848 Mar 12 10:28 mybin.a
-r-xr-xr-x 1 root root  160 Mar 12 10:28 mybin.x

Which in turn means different image digests.

I made a full repro here: https://github.com/tommyknows/rules_go_repro

To test this on linux, I've ran this in Docker:

docker run --entrypoint="" -v (pwd):/project --rm -it gcr.io/cloud-marketplace-containers/google/bazel /bin/bash

Am I doing something wrong? or is this a bug in rules_go? (or a known issue, and can it be fixed / worked around?)

Thanks!

[jayconrod]

I'm not entirely sure I see where the bug is here.

Paths can definitely be different on different platforms, even when the target platforms are the same. The k8-fastbuild-ST-5fa8105c1248 component of the path is generated by Bazel. rules_go has no control over that.

There's also a pretty good chance the .a files won't be reproducible unless you're very careful with -trimpath and stripping debug symbols.

I wouldn't expect copying image digests to source files to work reliably.

[tommyknows]

Thanks for your answer.

In order to get reproducible binaries (which is a requirement for a reproducible container image), the .a files need to be reproducible too, right?

So far the only thing we did is using the go_image rule, we didn't specify any "special" build rules like stripping debug symbols. And this has worked since we started using it ~1 year ago.

However, we've only ever used the files from the workspace, and not generated files like the one in my example.

I wouldn't expect copying image digests to source files to work reliably.

May I ask why?

The main issue here is including files that are not within the workspace, as my reproduction shows. I would expect that to "just work", though it is never as easy as that :-)

[tommyknows]

Just spent quite some time on this again because I was trying to get go_proto_library to work (reproducibly), until I realised that it has the same root cause: the "templated" - in this case generated pb.go - file.

This means that it's not possible to get reproducible builds if using protobuffers (except with some hacky workarounds).