[Community Action Requested] Survey: Managing binaries/dependencies in native projects

[lacasseio]

As part of our native effort, we want to better understand the use cases that native developers have with regards to managing binaries. Please comment on this issue with your stories, use cases, needs and wants regarding binaries that were built outside of Gradle ecosystem and that needs to be consumed inside your Gradle projects. When providing your feedback, it's important to make the distinction between what is required for your build compared to what is possible to do. We want to invest our initial effort where it will be the most beneficial for everyone. Thank you!

Guiding Questions

• What techniques/package management tools is your team using for handling binaries?
• What does the process look like? How do things end up on the machine?
• If you're mixing build systems (Gradle, Make, CMake, etc), how do you share dependencies?
• Do you use arbitrary packages/tarballs? If yes...
  • How often is it used in your code base?
  • What are some examples of those packages?
  • Are they installed/updated manually by the developer or is it automatically done by your build system?
  • Why not use package management tools?
  • What are the challenges you face with those arbitrary packages?
• If you're using the new C++ plugins...
  • What sucks when handling binaries from outside the build?
  • What sort of Gradle code have you written to handle these binaries?

Some of our thoughts

It's important to make a clear distinction between the producer and the consumer of a binary. A consumer should only need to express what is needed, e.g. I need Qt.Core version 5 or I need Boost:Asio. Decisions about which variant you would need should be handled by Gradle's dependency management engine via attributes--such as VS2013 vs. VS2019 ABI, or single-threaded vs multi-threaded, or x86 vs x86-64. With this in mind, for consumers, we think dependencies should look at simple as they do now for Java projects. This works for external dependencies published by Gradle, but for other kinds of dependencies, there's nothing built in.

A producer should know a lot more about a binary, such as where to find it and which attributes apply to it. For instance, binaries can be found in known locations (e.g. /usr/lib and /usr/include), in a structured format (e.g. homebrew, MacPorts, chocolatey), discoverable (e.g. Windows SDK via registry or vswhere.exe) or chaotic (seemingly random include folders and/or library files scatter across a system). Some of these binaries may come with additional metadata that describe the attributes Gradle understands. We think producers should be responsible for providing these details to Gradle. We have some experiments in this area for using CMake to build a library and Gradle consuming it.

We understand some of the frustration when dealing with libraries in a native project. We believe this frustration comes from the lack of identity for dependencies that can creep into a project. We want to avoid a situation where include paths, libraries or arbitrary files without identities are needed to build a project. We would like existing Gradle tools like gradle dependencies to show you identities so it's easier to reason about the build.

[zosrothko]

• What techniques/package management tools is your team using for handling binaries? git submodule for OpenSSL and installation zip for MySQL, PostsgreSQL, MongoDB, Redis.

• What does the process look like? How do things end up on the machine? A set of directories where one can found the includes and the link libraries

• If you're mixing build systems (Gradle, Make, CMake, etc), how do you share dependencies? Implicitly by naming the prebuild libraries paths (include dir and link lib dir)

• Do you use arbitrary packages/tarballs? If yes...

  • How often is it used in your code base? Always

  • What are some examples of those packages? OpenSSL, PostgreSQL, MySQL, MongoDB, Redis

  • Are they installed/updated manually by the developer or is it automatically done by your build system? Manually by the developper

  • Why not use package management tools? Those prebuild binaries are not available all at the same time thru a specific packager

  • What are the challenges you face with those arbitrary packages? Making automatic installation of those prebuild libraries on IC platforms like Travis/AppVeyor

• If you're using the new C++ plugins...

  • What sucks when handling binaries from outside the build? Different standards in naming the basename and path for release versus debug binaries

  • What sort of Gradle code have you written to handle these binaries?

    libs(PrebuiltLibraries) {
        WS2_32 {
            headers.srcDir WDKHome + "/Include/" + WDKVers + "/um/x86"
            binaries.withType(StaticLibraryBinary) {
                if (targetPlatform.operatingSystem.windows) {
                    staticLibraryFile = makePreBuildLibrary("WS2_32", targetPlatform)
                    println "staticLibraryFile=" + staticLibraryFile
                }
            }
        }
        def opensslHome = new File(rootDir, "openssl/build")
        def opensslBrewHome = new File('/usr/local/opt/openssl')
        def opensslLinuxHome = new File('/usr/lib/x86_64-linux-gnu')
        crypto {
            headers.srcDir "$opensslHome/include"
                binaries.withType(StaticLibraryBinary) {
                def libName = "foobar"
                if (buildType == buildTypes.debug) {
                    if (targetPlatform.name == 'win32') {
                        libName = 'libcryptod.lib'
                        staticLibraryFile = file("$opensslHome/win32/lib/debug/$libName")
                    } else if (targetPlatform.name == 'win64') {
                        libName = 'libcryptod.lib'
                        staticLibraryFile = file("$opensslHome/win64/lib/debug/$libName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        libName = 'libcryptod.a'
                        staticLibraryFile = file("$opensslBrewHome/lib/$libName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        libName = 'libcryptod.a'
                        staticLibraryFile = file("$opensslLinuxHome/$libName")
                    }
                } else
                if (buildType == buildTypes.release) {
                    if (targetPlatform.name == 'win32') {
                        libName = 'libcrypto.lib'
                        staticLibraryFile = file("$opensslHome/win32/lib/release/$libName")
                    } else if (targetPlatform.name == 'win64') {
                        libName = 'libcrypto.lib'
                        staticLibraryFile = file("$opensslHome/win64/lib/release/$libName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        libName = 'libcrypto.a'
                        staticLibraryFile = file("$opensslBrewHome/lib/$libName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        libName = 'libcrypto.a'
                        staticLibraryFile = file("$opensslLinuxHome/$libName")
                    }
                } else {
                    throw new GradleException("Unknown buildType" + buildType)
                }
            }
            binaries.withType(SharedLibraryBinary) {
                def dllName
                def linkName
                if (buildType == buildTypes.debug) {
                    if (targetPlatform.name == 'win32') {
                        dllName = 'libcryptod.dll'
                        linkName = 'libcryptod.lib'
                        sharedLibraryFile    = file("$opensslHome/win32/bin/debug$dllName")
                        sharedLibraryLinkFile = file("$opensslHome/win32/bin/debug/$linkName")  
                } else if (targetPlatform.name == 'win64') {
                        dllName = 'libcryptod.dll'
                        linkName = 'libcryptod.lib'
                        sharedLibraryFile    = file("$opensslHome/win64/bin/debug/$dllName")
                        sharedLibraryLinkFile = file("$opensslHome/win64/bin/debug/$linkName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        dllName = 'libcryptod.dylib'
                        linkName = 'libcryptod.dylib'
                        sharedLibraryFile    = file("$opensslBrewHome/lib/$dllName")
                        sharedLibraryLinkFile = file("$opensslBrewHome/lib/$linkName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        dllName = 'libcryptod.so'
                        linkName = 'libcryptod.so'
                        sharedLibraryFile    = file("$opensslLinuxHome/$dllName")
                        sharedLibraryLinkFile = file("$opensslLinuxHome/$linkName")
                    }
                } else
                if (buildType == buildTypes.release) {
                    if (targetPlatform.name == 'win32') {
                        dllName = 'libcrypto.dll'
                        linkName = 'libcrypto.lib'
                        sharedLibraryFile    = file("$opensslHome/win32/bin/release/$dllName")                  sharedLibraryLinkFile = file("$opensslHome/win32/bin/release/$linkName")
                    } else if (targetPlatform.name == 'win64') {
                        dllName = 'libcrypto.dll'
                        linkName = 'libcrypto.lib'
                        sharedLibraryFile    = file("$opensslHome/win64/bin/release/$dllName")
                        sharedLibraryLinkFile = file("$opensslHome/win64/bin/release/$linkName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        dllName = 'libcrypto.dylib'
                        linkName = 'libcrypto.dylib'
                        sharedLibraryFile    = file("$opensslBrewHome/lib/$dllName")
                        sharedLibraryLinkFile = file("$opensslBrewHome/lib/$linkName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        dllName = 'libcrypto.so'
                        linkName = 'libcrypto.so'
                        sharedLibraryFile    = file("$opensslLinuxHome/$dllName")
                        sharedLibraryLinkFile = file("$opensslLinuxHome/$linkName")
                    }
                } else {
                    throw new GradleException("Unknown buildType" + buildType)
                }
                }
        }
        ssl {
            headers.srcDir "$opensslHome/include"
                binaries.withType(StaticLibraryBinary) {
                def libName
                if (buildType == buildTypes.debug) {
                    if (targetPlatform.name == 'win32') {
                        libName = 'libssld.lib'
                        staticLibraryFile = file("$opensslHome/win32/lib/debug/$libName")
                    } else if (targetPlatform.name == 'win64') {
                        libName = 'libssld.lib'
                        staticLibraryFile = file("$opensslHome/win64/lib/debug/$libName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        libName = 'libssld.a'
                        staticLibraryFile = file("$opensslBrewHome/lib/$libName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        libName = 'libssld.a'
                        staticLibraryFile = file("$opensslLinuxHome/$libName")
                    }
                } else
                if (buildType == buildTypes.release) {
                    if (targetPlatform.name == 'win32') {
                        libName = 'libssl.lib'
                        staticLibraryFile = file("$opensslHome/win32/lib/release/$libName")
                    } else if (targetPlatform.name == 'win64') {
                        libName = 'libssl.lib'
                        staticLibraryFile = file("$opensslHome/win64/lib/release/$libName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        libName = 'libssl.a'
                        staticLibraryFile = file("$opensslBrewHome/lib/$libName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        libName = 'libssl.a'
                        staticLibraryFile = file("$opensslLinuxHome/$libName")
                    }
                } else {
                    throw new GradleException("Unknown buildType" + buildType)
                }
            }
            binaries.withType(SharedLibraryBinary) {
                def dllName
                def linkName
                if (buildType == buildTypes.debug) {
                    if (targetPlatform.name == 'win32') {
                        dllName = 'libssld.dll'
                        linkName = 'libssld.lib'
                        sharedLibraryFile    = file("$opensslHome/win32/bin/debug$dllName")
                        sharedLibraryLinkFile = file("$opensslHome/win32/bin/debug/$linkName")              } else if (targetPlatform.name == 'win64') {
                        dllName = 'libssld.dll'
                        linkName = 'libssld.lib'
                        sharedLibraryFile    = file("$opensslHome/win64/bin/debug/$dllName")                    sharedLibraryLinkFile = file("$opensslHome/win64/bin/debug/linkName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        dllName = 'libssld.dylib'
                        linkName = 'libssld.dylib'
                        sharedLibraryFile    = file("$opensslBrewHome/lib/$dllName")
                        sharedLibraryLinkFile = file("$opensslBrewHome/lib/$linkName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        dllName = 'libssld.so'
                        linkName = 'libssld.so'
                        sharedLibraryFile    = file("$opensslLinuxHome/$dllName")
                        sharedLibraryLinkFile = file("$opensslLinuxHome/$linkName")
                    }
                } else if (buildType == buildTypes.release) {
                    if (targetPlatform.name == 'win32') {
                        dllName = 'libssl.dll'
                        linkName = 'libssl.lib'
                        sharedLibraryFile    = file("$opensslHome/win32/bin/release/$dllName")                  sharedLibraryLinkFile = file("$opensslHome/win32/bin/release/$linkName")
                    } else if (targetPlatform.name == 'win64') {
                        dllName = 'libssl.dll'
                        linkName = 'libssl.lib'
                        sharedLibraryFile    = file("$opensslHome/win64/bin/release/$dllName")                  sharedLibraryLinkFile = file("$opensslHome/win64/bin/release/$linkName")
                    } else if (targetPlatform.operatingSystem.macOsX) {
                        dllName = 'libssl.dylib'
                        linkName = 'libssl.dylib'
                        sharedLibraryFile    = file("$opensslBrewHome/lib/$dllName")
                        sharedLibraryLinkFile = file("$opensslBrewHome/lib/$linkName")
                    } else if (targetPlatform.operatingSystem.linux) {
                        dllName = 'libssl.so'
                        linkName = 'libssl.so'
                        sharedLibraryFile    = file("$opensslLinuxHome/$dllName")
                        sharedLibraryLinkFile = file("$opensslLinuxHome/$linkName")
                    }
                } else {
                    throw new GradleException("Unknown buildType" + buildType)
                }
            }
        }
    }

[k-mack]

My responses are similar to @zosrothko's:

• What techniques/package management tools is your team using for handling binaries? Tarballs that contain the public API headers (i.e., the usual libname/include directory) and the link library variants (debug/release, shared/static) buried into a nested directory layout (e.g., libname/lib/os/arch/debug/shared). Despite the multitude of developer preferences within the organization, at the end of the day, every team can consume another team's tarball of binaries.

• What does the process look like? How do things end up on the machine? The process is usually pulling the tarballs down from a NAS or Artifactory instance. In the case of using Artifactory, some will publish the tarballs to a Maven repository and use Gradle/Maven/Ant&Ivy only for their dependency resolution skills. After unpacking the tarballs, a set of directories where one can find the headers and the link libraries ends up in the repository (but not checked in).

• If you're mixing build systems (Gradle, Make, CMake, etc), how do you share dependencies? Tarballs. If the project is really complex, we'll create a Docker container of everything needed to compile the project. Developers and CI can use that container to build the code.

• Do you use arbitrary packages/tarballs? If yes...

  • How often is it used in your code base? Always.

  • What are some examples of those packages? Boost.

  • Are they installed/updated manually by the developer or is it automatically done by your system? Automatically if using a Maven dependency resolution system; otherwise, it is done manually.

  • Why not use package management tools? Not all versions of libraries are available. Plus, most libraries are compiled with specific flags.

  • What are the challenges you face with those arbitrary packages? Integrating into continuous integration. We need to write boilerplate logic to pull down the right tarballs.

• If you're using the new C++ plugins...

  • What sucks when handling binaries from outside the build? if-else statements.

  • What sort of Gradle code have you written to handle these binaries? @zosrothko submitted a great example, which looks like it uses the software model. Here's a small sample from my usage with the new C++ plugins:

plugins {
    id "cpp-application"
}

def libBoostIncludePath = file("3rdparty/boost-1.60.0.2/include")

def linkLibs = [
    "linux:x86:debug:boost_program_options"     : file("3rdparty/boost-1.60.0.2/lib32/gcc4.8/libboost_program_options-debug.a"),
    "linux:x86:release:boost_program_options"   : file("3rdparty/boost-1.60.0.2/lib32/gcc4.8/libboost_program_options.a"),
    "linux:x86:debug:boost_system"              : file("3rdparty/boost-1.60.0.2/lib32/gcc4.8/libboost_system-debug.a"),
    "linux:x86:release:boost_system"            : file("3rdparty/boost-1.60.0.2/lib32/gcc4.8/libboost_system.a"),

    "linux:x86_64:debug:boost_program_options"  : file("3rdparty/boost-1.60.0.2/lib64/gcc4.8/libboost_program_options-debug.a"),
    "linux:x86_64:release:boost_program_options": file("3rdparty/boost-1.60.0.2/lib64/gcc4.8/libboost_program_options.a"),
    "linux:x86_64:debug:boost_system"           : file("3rdparty/boost-1.60.0.2/lib64/gcc4.8/libboost_system-debug.a"),
    "linux:x86_64:release:boost_system"         : file("3rdparty/boost-1.60.0.2/lib64/gcc4.8/libboost_system.a"),

    "windows:x86:debug:boost_program_options"   : file("3rdparty/boost-1.60.0.2/lib32/vs2015/boost_program_options-debug.lib"),
    "windows:x86:release:boost_program_options" : file("3rdparty/boost-1.60.0.2/lib32/vs2015/boost_program_options.lib"),
    "windows:x86:debug:boost_system"            : file("3rdparty/boost-1.60.0.2/lib32/vs2015/boost_system-debug.lib"),
    "windows:x86:release:boost_system"          : file("3rdparty/boost-1.60.0.2/lib32/vs2015/boost_system.lib"),

    "windows:x86_64:debug:boost_program_options"    : file("3rdparty/boost-1.60.0.2/lib64/vs2015/boost_program_options-debug.lib"),
    "windows:x86_64:release:boost_program_options"  : file("3rdparty/boost-1.60.0.2/lib64/vs2015/boost_program_options.lib"),
    "windows:x86_64:debug:boost_system"             : file("3rdparty/boost-1.60.0.2/lib64/vs2015/boost_system-debug.lib"),
    "windows:x86_64:release:boost_system"           : file("3rdparty/boost-1.60.0.2/lib64/vs2015/boost_system.lib"),
]

application {
    targetMachines = [
            machines.linux.x86,
            machines.linux.x86_64,
            machines.windows.x86,
            machines.windows.x86_64,
    ]

    binaries.whenElementFinalized { binary ->
        def arch = binary.targetMachine.architecture.name == MachineArchitecture.X86_64 ? "x86_64" : "x86"
        def os = binary.targetMachine.operatingSystemFamily.windows ? "windows" : "linux"
        def buildType = binary.optimized ? "release" : "debug"
        project.dependencies {
            add(binary.includePathConfiguration.name, files(libBoostIncludePath))
            add(binary.linkLibraries.name, files(linkLibs["${os}:${arch}:${buildType}:boost_program_options"]))
            add(binary.linkLibraries.name, files(linkLibs["${os}:${arch}:${buildType}:boost_system"]))
        }
    }
}

[lacasseio]

Thanks a lot @zosrothko and @k-mack for your responses. It seems tarball of the library is the usual way to consume dependencies for your use cases.

If Gradle was able to consume dependencies from various source (different package manager) would you envision fetching some dependencies from those package manager instead of tarball or the issue that "most libraries are compiled with specific flags" is too much of an issue for those package manager binaries?

Those samples are very helpful to see the pain you are going through. In my previous job, we didn't have much-prebuilt binaries, but I do think it's one of the big issue that need to be solved sooner.

[lacasseio]

Great feedback, it seems the most important use case identified here and by other channel is consuming prebuilt binaries from arbitrary packages. It should be possible to accomplish this now, a demonstration issue was open to create a sample to guide users. We will consider more out-of-the-box features once the sample have been written.