Feature request: Provide alternative library install strategies in InstallExecutable

[urdak]

For large multi-project builds copying all libraries that the executable depends on produces significant overhead in both disk space usage and execution time.

Expected Behavior

InstallExecutable task should be able to construct a library path from dependencies without copying them to a dedicated directory.

Current Behavior

InstallExecutable task copies all dependent libraries to a directory and adds that directory to the path.

Context

I maintain a large multi-project gradle build. Most of the projects are C++ and are currently using the old model-based plugin. A self-written test plugin creates a run script that adds all library dependency directories to LD_LIBRARY_PATH. I would like to migrate to the new C++ plugins. One of the major issues preventing this migration is the fact that the library dependency chains take too much disk space when copied for each test.

Possible Solutions

• Construct library path by specifying original directory locations of library dependencies.
• Create symlinks to dependencies in the library directory instead of copying them (if supported by the OS).
• Add a way to provide a custom InstallExecutable task implementation instead of the default one. This would also enable additional customization for the executable run script.

[lacasseio]

Thanks @urdak for the feedback. Sadly, not everything is properly explained in the documentation and it's also hard to explain the everything, especially the work in progress aspect. The InstallExecutable task is one way to prepare a distributable package for your application. It's certainly not the best nor the only way to do it. What you are asking makes perfect sense and should be possible to customize with the present version. One downside though is it's not possible to remove the current InstallExecutable task but you can certainly disable it.

What would be your ideal install task for your project? We can start from there and create a sample for that specific use case. Then we can look what should be improved in the plugin to make it easier to use for that use case.

On another note, we are always searching for great user like you to drive the adoption forward, especially the migration from the software model plugin to the new C++ plugins. Send me a quick description of your project as well as your top 5 issues that would unblock your migration (most important) or that would make Gradle easier to use in your project at native-team@gradle.com

[urdak]

Hi @lacasseio, thank you for the quick response.

Here's a rough representation of what the current implementation does:

• Constructs LD_LIBRARY_PATH. This is done from a project extension that is populated from the model using binaries.withType(SharedLibraryBinarySpec).
• Constructs PATH. Same as LD_LIBRARY_PATH, using binaries.withType(NativeExecutableBinarySpec).
• Exports multiple environment variables required for the test (including PATH and LD_LIBRARY_PATH).
• Sets ulimit.
• Optionally can use a debug tool to run the executable.
• Sets up --gtest_output.
• Sets up line buffering using 'stdbuf'.
• Saves the output to a file and gzip's it. Prints where the output can be found.
• If a core dump was generated, displays available debug information by running gdb on the core.

This is all done by concatenating a bunch of strings and saving them to a bash file (similar to what InstallExecutable does).

Regarding other issues: I would love to provide more feedback. I will try to get back to you via email some time in January.

[urdak]

Hi @lacasseio,

I'm still wondering what are the alternatives to InstallExecutable for running a unit test. Do I create my own tasks on top of compiling/linking provided by the unit test plugin and disable InstallExecutable and RunTestExecutable tasks?

Sorry I didn't have time to provide more feedback via email, I will do that as soon as I can.

[Alex-Vol-SV]

Hi @lacasseio

I have a very similar use case that I solved in the Software Model plugins by doing an almost identical implementation to what @urdak described. I have created my own flavor of Boost-Test plugin extrapolated from the GoogleTest plugin. I disabled the InstallExecutable and the dependency to RunTestExecutable and substituted the runExecutable with a generated script that I create modeled after what InstallExecutable was doing. So, what @urdak describes is very doable.

Currently I am porting this solution to the new C++ library plugins by forking the code in the cpp-unit-test-plugin to my own construction for a few reasons:

1. I have more than one unit test suite in my implementation, in fact I have hundreds of test suite executables. One library hundreds of test cases. Why is that? Because the library is really large with so much functionality that doing in depth testing must be partitioned in many suites.
2. I require to properly isolate uses of mocking by stubbing out code. The way unit tests do that is by declaring methods just like the production runtime code and relying on the behavior of the linker that will simply omit resolving symbols already provided by the core executable.
3. I also require building both release and debug mode executables for unit test as well as publishing to the Artifact repository. The reason for doing that is so that developers can fetch the appropriate compiled debug mode library when they use it in a dependency without having to compile it locally. We have 80+ code repositories all producing over 100 libraries that are used in a large application with multiple executables. Building the complete application set in release and debug mode takes over 16 hours time and over a terabyte of objects with symbols is created and linked together.
4. For the same reason I changed how the unit test executable is built, with shared libraries I set the testedComponent as a library dependency picking the matching variant (debug/release) as needed. I do not relink the library into the test executable. This library by itself is about 450MB or code and symbols, assuming most of the code would end up linked in each test executable I would be looking for about a gigabyte of storage per test executable with one release and one debug executable.

Imagine copying a substantial set of dependencies over hundredfold. I would need terabytes of disk space and time to create and build tests for a single repository build.