Open amz-shahji opened 5 years ago
Publishing binaries as there were built by Gradle is one way to do it. However, you can achieve the same result without publishing the binaries inside a maven or nexus repository. At the moment, the moment, publishing binaries through Gradle requires a lot of internal API. The sample you are referring was created to unblock a particular client. I wouldn't suggest going down that route at the moment. I agree we will provide public API to aid with the publication of 3rd party binaries. We don't have a clear roadmap on when such API will be part of Gradle.
An alternative solution is to consume the prebuilt binaries as they are published by the vendor and expose them in Gradle by linking the binaries to the dependency management. This solution doesn't use any internal API. The concept goes as follow:
1) If the vendor publishes the package as a zip, have a set of tasks to download and extract the zip and link those tasks in the dependency management. See prebuilt-sdk
sample.
2) If the vendor publishes the package as an installer, either have Gradle run the installer to "extract" the binaries (complicated, but can be easy for the self-extracting executable on Windows) or massage the installer into a zip that you expose in some way followed by option (1) (easier). This option was successful in the past for dealing with Windows SDK and DDK.
3) If the vendor publishes the package inside some package manager such as Homebrew, Yum, MacPorts, Chocolatey, etc., have Gradle locate the binaries in known locations for those package manager. You can have Gradle actively install them.
All other scenarios that we can think of at the moment are a variation of those solutions. We understand those solutions may be more of the same of what you already have and aren't ideal. To move this story forward, could you document your use case here? The questions we would be interested in are:
@lacasseio
An alternative solution is to consume the prebuilt binaries as they are published by the vendor and expose them in Gradle by linking the binaries to the dependency management.
This will require managing this dependency at every level in the project hierarchy. Any solution outside of Gradle will either not support transitive dependency or will end up doing lot of work what Gradle already does to manage the lifecycle.
How the vendor of your prebuilt binaries publishing them? What format is used? Archive? Individual binaries? Installer? Package (for package managers)? How are they served? Http? Ftp? NFS? Package manager? Checked inside source control? Well-known location on the file system?
Since we are our own vendor, the format is always headers and prebuilt binaries (not binary, i.e. not necessarily just one). At the moment, they are being served directly from our repository as part of one really huge monolithic blob.
What kind of variation of the binaries are available? Debug/release? With some features enabled/disabled?
This varies widely. Usually, at least debug and release. In some cases, we have feature based binaries like with different log levels, whether the binaries where built with asm enabled or not, and in some cases, built with optimizations enabled but in-lining disabled to support debuggers.
Are the variations of the binaries part of the same "package" provided by the vendor or they are scattered around in multiple places that need to be reconciled.
Usually they are. For 9/10 use cases, what we have is one monolithic blob. Consumer chooses what's to be consumed.
Are you required to consume a range of prebuilt version throughout all the project?
Always. Teams move at different pace and some can upgrade earlier then others. Plus, some can even walk back if they find some feature that they need broken in newer release.
Out of all those use case that we gather with the previous questions, which one is the most used pattern?
I want to publish a module with headers and multiple libraries per variant and pick the variant at consumption time. For e.g. OpenSSL comes with a few headers and at least two libraries each for debug and release. For half the use cases, only one or the other library is used, so ability to define multiple variants is important here (because linking both libraries would fail).
Variant 1 - Headers + Debug library A Variant 2 - Headers + Debug library B Variant 3 - Headers + Release library A Variant 4 - Headers + Release library B Variant 5 - Headers + Profile library A Variant 6 - Headers + Performance library B and so on.
Our use cases fall in multiple different buckets -
At the moment, most, if not all, these prebuilt binaries are committed to our internal repository (and no we are not using Git).
We can make a decent progress with minimal support from within Gradle without digressing too far from long term solution. If Gradle can support a plugin to define paths to debug and release library/binary in the DSL, and this plugin is an extension to CppStaticLibrary (and CppSharedLibrary) and just wire it in such a way to avoid compilation, we could be in business. The plugin will explicitly set the output of the compile task with the input from the DSL and mark it realized. This the quickest path to get headers + one binary (with transitive dependency) into the pipeline. If this can be realized as an external plugin, tell us how. Do you see any immediate pitfalls. Its understood that this is not a long term solution but will get up over the immediate bump.
Hi @amz-shahji
Did u eventually find a way to publish prebuilt binaries?
Thanks
@awaizman1 Publishing pre-build binaries isn't possible out of the box, but if you have a specific use-case, I can write you a sample to ensure you can progress with your Gradle native journey. :-)
Appreciate it @lacasseio,
My use case is very simple: I have prebuilt binaries (in the form of .dll + .lib, .so) and public headers folder. The files either obtained from 3rd party vendor or manually built by me according to 3rd party build procedure. Now I want to publish those binaries and headers (for win and Linux) to a private maven repo (where I publish other gradle cpp projects to).
Thanks.
Hi @lacasseio
could you reference me for such sample please?
thanks
Sorry for the delay in responding, I will write up some quick sample tomorrow for those use cases.
@awaizman1 Here is a sample that should put you on the right track for now. The samples try to be a bit more flexible than just configuring a bunch of Configuration
objects by modelling the core concept used in Nokee. A quick rundown of the concept goes as follow:
Keep in mind the sample simplifies greatly the problem and how variants are handled. Most of the boilerplate code in this sample should go away in the next few Nokee releases if you use the plugins. With the Gradle core plugins, you will have to keep that boilerplate code and most likely wrap it inside a custom plugin.
Note that the sample doesn't try to solve the issue with partial publishing (when you build binaries on different systems and want to publish only a subset of the variant on each machine). The sample assumes you have all the binaries built and ready to be published.
Don't hesitate to ask more questions, I understand the core concept can be confusing at first. I'm still trying to find a good way to explain them properly. It's a powerful feature that leans a bit toward being too abstract.
Need an easier, end-user friendly way of publishing prebuilt binaries. There are examples here and here that can be used create a version of the plugin which works but something part of the API would be ideal.
Expected Behavior
New plugin, say "cpp-prebuilt", with option for profiles/configuration/platform and all the other necessary jingle-bells.
Current Behavior
Home grown solution works but isn't extendable and is incomplete in many ways. There isn't one in the API, as of yet.
Context
We publish 150+ prebuilt binaries that are produced by build system using anything but gradle. These are distributed as prebuilt binaries (headers, static libs, dynamic dlls) for consumption. These need to be published using the same "signature" as if these were the output of a successful gradle build so they can be consumed downstream identically.
Steps to Reproduce (for bugs)
Publish any library project in native-samples repo and try consuming that in corresponding application.
Your Environment