Publishing mutiple prebuilt binaries as one artifact and consume selectively

[amz-shahji]

Dependent on #900

Certain publicly available projects (and internal too) generate multiple binaries as output and should be published as a single artifact. Known examples - OpenSSL (2), Qt (50+), Wwise (75+), zlib, etc.

Expected Behavior

These prebuilt binaries can be published as a single artifact and be consumed by cherry-picking which specific one is required.

Current Behavior

Each prebuilt binary has to be published individually which in turn publishes the headers artifact multiple times. Linking as multiple api/implementation of the same headers causes number of issues -

• Symbol duplication for all the inline functions coming for two sets of same headers
• Symbol duplication where one library happens to be a superset of the other
• Wrong include file paths because two libraries where built using the same headers but different macros that enable/disable part of the same header (and thus changing the class prototypes)
• ... and the list goes on....

Context

Example use cases -

• OpenSSL includes two libs libeay32.lib and ssleay32.lib. Projects can use any combination of these two libraries, either one or both.

• Qt includes 50+ different libraries. Projects can cherry pick what they wish to use.

• GoogleTest includes two binaries - gmock and gtest.

• zlib is generated as MDd and MTd flavors. Consumer cherry picks based on project needs.

Note the user, for the most part, never wants to link all the libs automatically. For many of these use cases, it is likely to be a disaster.

[lacasseio]

I don't think the solution here is to publish as one artifact but ensure the publication link conceptually equivalent artifact together to avoid duplication which leads to what you are referring. Out of what is mentioned, there are two competing features required:

1. Support for multiple publishing multiple components per project that can be consumed individually (Qt, OpenSSL, and GoogleTest)
2. Support for custom variants of a component to be matched automatically or disambiguated through rules (zlib).

Both should be able to be accomplished at the moment but may require internal APIs to be used. As a first and most powerful first step could be aiming toward public APIs available through the dependency engine so these are easily achievable with provided sample and documentation.

@gradle/dependency-management Is this something easy-ish to accomplish at the moment and is the idea of opening up public API from the dependency management side a better idea than the alternative to model something over the internal APIs?

[bigdaz]

@lacasseio I'm afraid I don't really comprehend the problem well enough to help. As you noted, the case where 2 variants are published with only 1 chosen is pretty straightforward, even if it's not simple to achieve.

For the OpenSSL/Qt case, it sounds like these are independent libraries that share the same header files, but I'm guessing it's not so simple. I don't think that publishing these all as a single module is necessarily the right way to go. Instead, I'd investigate publishing the API as a module and having the separate library modules depend on that API.

[lacasseio]

Thank @bigdaz, in short, it would be possible to achieve this at the moment but wouldn't be as simple.

[amz-shahji]

I don't think that publishing these all as a single module is necessarily the right way to go.

Unfortunately, we might not have an option, especially, for the publicly available projects (untouched for years) and large codebase where having to make this choice is last thing on the list. In some cases, the situation is even worse. The code was built on one blue moon night and never again. All we have are the binaries that"work". Plus, as I mentioned on one another thread, at times having a single large monolithic artifact is considered a feature (like in the case of QT).

The list of such projects is rather long. Unfortunately, many are internal and proprietary so I can't really list them all here. One common thing among many is "We", as a team, don't own them. We are downstream consumers and the projects generally have their own build systems that generate these binaries. Our projects are consuming them as they are and the first thing we would like to do consume them as is. Its very important that we stand up a parallel build system and stabilize it before we drop support for the existing one. And making supporting changes for existing build system is not always trivial (or in some cases even possible).

@lacasseio mentioned something about components. Could you elaborate more on that and how that fits. Does this model support notion of exported macros #755 and the 'messy include paths' #869

[lacasseio]

As with #900 we don't have good use cases right now with regards to publishing 3rd party prebuilt library. We are unlikely to move forward with such feature in the short time. I suggest looking at this comment for a more flexible solution while we gather the use case for this issue.

With regards to the components solution. I think it's a solution down the road that could be done now but isn't documented enough. Given the comment on issue #900, I feel like it wouldn't be a viable solution in the short term because of the requirements to use internal APIs.