Spike Conan package resolution

lacasseio commented 2 years ago

The leading idea for resolving Conan package in Gradle is to use Conan metadata artifacts together with the Gradle artifact transform. Basically, the metadata artifacts are transformed into the installed Conan package. To resolve a Conan metadata artifact, we will use a Gradle module metadata. Our current understanding of the Gradle dependency engine should allow this process to work not only for Conan but for other package managers. This issue serves to derisk the approach.

Open Questions

How to locate the artifacts of interest within the package? The recipes don't seem to clearly point to the interesting part of the package (headers, link library, runtime library, etc.). We may have to settle on an intermediate consumer, e.g. pkgconfig.

lacasseio commented 2 years ago

It seems we could use the TXT generator. The JSON generator seems to be unstable. The Pkgconfig generator is interesting simply because the capability is reusable in other contexts (such as QT packages).

lacasseio commented 2 years ago

We made some great progress on this spike. The goal is for those third-party dependencies to behave like Gradle dependencies so we can use features like dynamic version, dependency substitutions, artifact transforms, etc. Not all third-party dependency managers are created equals so the mapping may not be as straightforward. With Conan, there are three central models: recipes, packages and components.

Recipes

The recipe is what users act upon. They need to specify some options, i.e. OS, architecture, with*, etc. that guide the installation of the packages. Recipes specify 2 types of transitive dependencies, e.g. require in Conan terms: the dependencies for building the source code and the dependencies of the source code. For Gradle, we only care about the dependencies of the source code.

Packages

The packages are what end up being installed. They are a cross-cut representation of all the selected options. There is an important distinction between Gradle and Conan to note. Conan allows users to inject configuration to the recipe which Gradle only allows pulling. The difference in thinking is Gradle dependency metadata should be immutable (predefined) while Conan is more on the fluid side where you can force certain behaviour such as using a different toolchain to compile the source resulting in a completely different meaning with respect to the provided metadata. Regardless, it doesn't mean Gradle cannot support configuration injection. We could inject additional configuration variants by using dependency manipulation rules on the consumer side.

Another important thing to note is how a package can represent a set of libraries. In a straightforward mapping, using packages as modules means depending on all libraries at the same time. For example, depending on boost means we depend on all 60-something libraries which is usually not what the user wants but may be useful for quick hacks. What users mostly want is depending on the individual libraries. Sadly, we haven't found a way to request all available components without first installing the packages.

Components

Finally, the packages contain one or multiple components which map to the individual libraries. Listing the available components seems to only be available at the package_info() stage which happens after a package is built and installed.

We also found that not all Conan commands are created equals. Generally, the --json flags provide a lot more information than the standard output or generator files.

Mapping to Gradle metadata

Since we need to build and install Conan recipes to access all component listing and their dependencies, the mapping need to account for the least amount of work when resolving to avoid installing the entire Conan repository locally :-) We also want to avoid globing the same components under different Gradle modules as we would lose the possibility to manipulate the transitive dependencies. Here is the proposition:

io.conan.<recipe>:<component>:<version>: When the group includes the recipe name, the module name will map to the Conan component. The metadata will expose, if available, the transitive dependencies of the component.
io.conan:<recipe>:<version>: When the recipe is the module name, the dependency will resolve to all components (through transitive dependencies).

As for mapping the options, we have two approaches: pure Gradle attributes or serialized options in the module name. Regardless of the approach, we end up using, the dependency declaration will be the same, e.g. we will rely on Gradle attributes and capabilities. For example, suppose we want boost recipe and math component with fPIC options to False the dependency declaration would look something like this:

repositories {
    conanCenter()
}

dependencies {
    implementation('io.conan.boost:math:1.77.0') {
        attributes {
            attribute(Attribute.of('fPIC', Boolean), false)
        }
    }
}

Implementation Details

The implementation relies on mapping the Conan metadata into Gradle metadata as Maven-compatible layout.

Listing the versions will rely on looking up the config.yml file directly in the GitHub repository for Conan Center index, i.e. https://raw.githubusercontent.com/conan-io/conan-center-index/master/recipes/abseil/config.yml. We are unsure how we will support listing for third-party Conan repository. The reason for relying on the GitHub repository is how much slow the conan search command is. During our testing, it would often take minutes to search the versions of a package and would often timeout.

For resolving the package, we will use conan info to gather the recipe's transitive dependency and conan inspect to find all available options, accepted values and default values. Finally, conan install is required to get the list of components and their transitive dependencies.

In terms of compatibility with Gradle plugins, we will most likely develop the plugin as a standalone capability plugin, i.e. dev.nokee.conan-support. We intend to use its own set of base attributes which will require some artifact transformation to work with either core Gradle native support or Nokee native support. Of course, we will offer Nokee integration out-of-the-box. This decision is to avoid tieing Conan support to strictly building software but allow any plugins to use our Conan support for their own needs and purpose.

lacasseio commented 2 years ago

One shortcoming of Conan that users have to be aware of is the lack of distinction between each component's headers. This shortcoming extends throughout the native ecosystem, not strictly Conan. For example, depending on any component of a recipe will often pull all of the boost's headers into scope meaning with a single dependency to boost::math users can now include boost::asio headers without any explicit dependencies to boost::asio. This boils down mostly to the pattern of putting all includes into a single include directory. We won't try to fix this issue at first but we could roll out in the future an isolation concept that either try to isolate the headers per component or open the possibility for users to specify isolation rules.

nokeedev / gradle-native