Reproducible C++ builds

[jamie-walker]

I've found two issues when trying to make a cpp-library build reproducible (i.e. produce the same binary output from the same source files):

1. The order of objectFiles in LinkerSpec is not deterministic. In the case where there is more than one input file, the linker may link the objects in a different order on different build hosts. This results in a different output. Probably sorting the List before invoking the compiler is the solution? Not sure whether this should be in LinkerSpec or in GccLinkerArgsTransformer or if there is workaround I can use for now without patching the gradle source?

2. I'd like to add compiler args like "-fdebug-prefix-map=${project.rootdir}=." to remove the project directory from the debug symbols, but this changes the build cache key due to the absolute path to the rootdir and defeats caching. I think I'd like a way of adding CommandLineArgumentProviders to the compile tasks somehow. I can work around this by forcing debuggable=false on the compile tasks instead, but this removes all debugging symbols from the resulting library which is not great.

Context

I'm using gradle to build a small jni shared library that is included inside the jar file of a java-library. Unfortunately this java library is included over 100 other subprojects. Because the .so file changes across build machines, the runtime classpath of the unit tests changes and stops the build cache from working.

Your Environment

I'm using gradle 6.3 on Linux.

[lacasseio]

Thanks for opening this issue. Your points are very good. The object files ordering should be constant between builds. For the -fdebug-prefix-map flag, we should add modelling around this to include the behaviour in the caching key as opposed to the flag itself. It seems the modelling would be something like: no prefix vs prefix from the root directory.

My feeling would be to prefix from the project directory, is there a reason you are prefixing from the root directory?

[jamie-walker]

I haven't thought about it too much or done any real testing here, but I thought rootDir made more sense in the context of a multi-project build: if you try to debug a binary that is linked from multiple other sub-projects then you hopefully only need to point the debugger at the rootDir in order for it to find all the source files. If the paths are relative to each project instead, it could get very confusing if there are two source files with the same name in different projects.

[lacasseio]

Good point, that make sense. The best solution right now, which is not great would be to ensure all paths are the same for all machines. You could mount the sync folder to a known location and build from there. On Windows, you can mount a folder to a virtual drive to create a common path. It's not the best, but it work around the issue for now.