Publishing prebuilt native binaries

[amz-shahji]

Need an easier, end-user friendly way of publishing prebuilt binaries. There are examples here and here that can be used create a version of the plugin which works but something part of the API would be ideal.

Expected Behavior

New plugin, say "cpp-prebuilt", with option for profiles/configuration/platform and all the other necessary jingle-bells.

Current Behavior

Home grown solution works but isn't extendable and is incomplete in many ways. There isn't one in the API, as of yet.

Context

We publish 150+ prebuilt binaries that are produced by build system using anything but gradle. These are distributed as prebuilt binaries (headers, static libs, dynamic dlls) for consumption. These need to be published using the same "signature" as if these were the output of a successful gradle build so they can be consumed downstream identically.

Steps to Reproduce (for bugs)

Publish any library project in native-samples repo and try consuming that in corresponding application.

Your Environment

• Build scan URL:

[lacasseio]

Publishing binaries as there were built by Gradle is one way to do it. However, you can achieve the same result without publishing the binaries inside a maven or nexus repository. At the moment, the moment, publishing binaries through Gradle requires a lot of internal API. The sample you are referring was created to unblock a particular client. I wouldn't suggest going down that route at the moment. I agree we will provide public API to aid with the publication of 3rd party binaries. We don't have a clear roadmap on when such API will be part of Gradle.

An alternative solution is to consume the prebuilt binaries as they are published by the vendor and expose them in Gradle by linking the binaries to the dependency management. This solution doesn't use any internal API. The concept goes as follow: 1) If the vendor publishes the package as a zip, have a set of tasks to download and extract the zip and link those tasks in the dependency management. See prebuilt-sdk sample. 2) If the vendor publishes the package as an installer, either have Gradle run the installer to "extract" the binaries (complicated, but can be easy for the self-extracting executable on Windows) or massage the installer into a zip that you expose in some way followed by option (1) (easier). This option was successful in the past for dealing with Windows SDK and DDK. 3) If the vendor publishes the package inside some package manager such as Homebrew, Yum, MacPorts, Chocolatey, etc., have Gradle locate the binaries in known locations for those package manager. You can have Gradle actively install them.

All other scenarios that we can think of at the moment are a variation of those solutions. We understand those solutions may be more of the same of what you already have and aren't ideal. To move this story forward, could you document your use case here? The questions we would be interested in are:

• How the vendor of your prebuilt binaries publishing them?
  • What format is used? Archive? Individual binaries? Installer? Package (for package managers)?
  • How are they served? Http? Ftp? NFS? Package manager? Checked inside source control? Well-known location on the file system?
• What kind of variation of the binaries are available? Debug/release? With some features enabled/disabled?
• Are the variations of the binaries part of the same "package" provided by the vendor or they are scattered around in multiple places that need to be reconciled. A better understanding of what I'm talking here is prebuilt Boost binaries. They are mostly in package managers, but each operating system family has its own package manager of choice. On Windows, the Boost prebuilt binaries are published on source forge as an installer for each Visual Studio version.
• Are you required to consume a range of prebuilt version throughout all the project? For example, some project needs Boost 1.63 while others use Boost 1.65, etc.
• Out of all those use case that we gather with the previous questions, which one is the most used pattern? We will try to solve the most important first.

[amz-shahji]

@lacasseio

An alternative solution is to consume the prebuilt binaries as they are published by the vendor and expose them in Gradle by linking the binaries to the dependency management.

This will require managing this dependency at every level in the project hierarchy. Any solution outside of Gradle will either not support transitive dependency or will end up doing lot of work what Gradle already does to manage the lifecycle.

How the vendor of your prebuilt binaries publishing them? What format is used? Archive? Individual binaries? Installer? Package (for package managers)? How are they served? Http? Ftp? NFS? Package manager? Checked inside source control? Well-known location on the file system?

Since we are our own vendor, the format is always headers and prebuilt binaries (not binary, i.e. not necessarily just one). At the moment, they are being served directly from our repository as part of one really huge monolithic blob.

What kind of variation of the binaries are available? Debug/release? With some features enabled/disabled?

This varies widely. Usually, at least debug and release. In some cases, we have feature based binaries like with different log levels, whether the binaries where built with asm enabled or not, and in some cases, built with optimizations enabled but in-lining disabled to support debuggers.

Are the variations of the binaries part of the same "package" provided by the vendor or they are scattered around in multiple places that need to be reconciled.

Usually they are. For 9/10 use cases, what we have is one monolithic blob. Consumer chooses what's to be consumed.

Are you required to consume a range of prebuilt version throughout all the project?

Always. Teams move at different pace and some can upgrade earlier then others. Plus, some can even walk back if they find some feature that they need broken in newer release.

Out of all those use case that we gather with the previous questions, which one is the most used pattern?

I want to publish a module with headers and multiple libraries per variant and pick the variant at consumption time. For e.g. OpenSSL comes with a few headers and at least two libraries each for debug and release. For half the use cases, only one or the other library is used, so ability to define multiple variants is important here (because linking both libraries would fail).

Variant 1 - Headers + Debug library A Variant 2 - Headers + Debug library B Variant 3 - Headers + Release library A Variant 4 - Headers + Release library B Variant 5 - Headers + Profile library A Variant 6 - Headers + Performance library B and so on.

Our use cases fall in multiple different buckets -

• We are our own vendor. We massage whatever is external to fit our build flow and make it available in our own repository. Our setup do not allow reaching out to download/install any external dependencies.
• We take ownership of certain publicly available project, make some tweaks and use it in our projects. These are usually minor tweaks to support either some unsupported platform or fix some minor bug and we don't want to upgrade to next available version. In either and all cases, we use the build system that comes with this inherited project. These build systems generally require additional dependencies and setup (like MinGW, MYSYS, CMake, autotools, etc). We don't expect every developer on the team to understand and know how to use these systems.
• We inherited a project from some other team within the organization and they don't use Gradle. In many cases, we don't even build their project (primarily because that requires a environment setup which in many cases is proprietary to the team). All we inherit is an archive (usually in proprietary format) that is unpacked and made available for our project to use.
• We license technology from third party vendors that only provide us with binaries. The format can vary widely. Our usual process is to do the necessary manual work to make it compatible within our system so they can be consumed by our projects.
• If some library was available only as an installer, we manually install it once and extract what's necessary to built. If that doesn't work, we modify our build machine image to pre-install such dependency.
• I am certainly missing more use cases but above ones should give you a good idea. We publish close to 200 modules and you can say, I am not familiar (how they were generated or how they are consumed) with all of them.

At the moment, most, if not all, these prebuilt binaries are committed to our internal repository (and no we are not using Git).

We can make a decent progress with minimal support from within Gradle without digressing too far from long term solution. If Gradle can support a plugin to define paths to debug and release library/binary in the DSL, and this plugin is an extension to CppStaticLibrary (and CppSharedLibrary) and just wire it in such a way to avoid compilation, we could be in business. The plugin will explicitly set the output of the compile task with the input from the DSL and mark it realized. This the quickest path to get headers + one binary (with transitive dependency) into the pipeline. If this can be realized as an external plugin, tell us how. Do you see any immediate pitfalls. Its understood that this is not a long term solution but will get up over the immediate bump.

[awaizman1]

Hi @amz-shahji

Did u eventually find a way to publish prebuilt binaries?

Thanks

[lacasseio]

@awaizman1 Publishing pre-build binaries isn't possible out of the box, but if you have a specific use-case, I can write you a sample to ensure you can progress with your Gradle native journey. :-)

[awaizman1]

Appreciate it @lacasseio,

My use case is very simple: I have prebuilt binaries (in the form of .dll + .lib, .so) and public headers folder. The files either obtained from 3rd party vendor or manually built by me according to 3rd party build procedure. Now I want to publish those binaries and headers (for win and Linux) to a private maven repo (where I publish other gradle cpp projects to).

Thanks.

[awaizman1]

Hi @lacasseio

could you reference me for such sample please?

thanks

[lacasseio]

Sorry for the delay in responding, I will write up some quick sample tomorrow for those use cases.

[lacasseio]

@awaizman1 Here is a sample that should put you on the right track for now. The samples try to be a bit more flexible than just configuring a bunch of Configuration objects by modelling the core concept used in Nokee. A quick rundown of the concept goes as follow:

• Each variant is defined as a cross-cutting slice of the possible values for each dimension.
• A dimension is a logical grouping of some values where at most one value can be selected per variant, for example, the dimension operating system family can include Linux, macOS, Windows, iOS, Android, etc as their values but only one would be selected for a particular variant. Other dimensions include machine architecture, build type and binary linkage.
• Each value selected per-dimension are grouped together to form the build variant.
• For each build variant, we create the necessary configuration to allow consumers to resolve the produced binaries.

Keep in mind the sample simplifies greatly the problem and how variants are handled. Most of the boilerplate code in this sample should go away in the next few Nokee releases if you use the plugins. With the Gradle core plugins, you will have to keep that boilerplate code and most likely wrap it inside a custom plugin.

Note that the sample doesn't try to solve the issue with partial publishing (when you build binaries on different systems and want to publish only a subset of the variant on each machine). The sample assumes you have all the binaries built and ready to be published.

[lacasseio]

Don't hesitate to ask more questions, I understand the core concept can be confusing at first. I'm still trying to find a good way to explain them properly. It's a powerful feature that leans a bit toward being too abstract.