search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.1k stars 960 forks source link

[question] How to depend on a specific Git commit or branch of a project #16309

Open tttapa opened 3 months ago

tttapa commented 3 months ago

What is your question?

Hi,

Is there any way to depend on a specific commit or branch of a package on Conan Center? E.g. similar to what Python's Pip does: https://pip.pypa.io/en/stable/topics/vcs-support/: MyProject @ git+https://git.example.com/MyProject.git@da39a3ee5e6b4b0d3255bfef95601890afd80709.

Use case:
Some packages (like Eigen) may go years without releasing a new version. I cannot use the latest release, because it is too old and I need some bug fixes that were added in later commits to the master branch.

What I'd like to be able to do is the following:

Creating my own "custom-eigen" package is not an option, because then a user could end up with a build tree containing both the official "eigen" package and my "custom-eigen" package, which would conflict.

Looking at the Conan documentation and the questions here on GitHub, I doubt this can be done exactly as described, but what are the alternatives? Which approach would you recommend for such a use case?

Thank you!

Have you read the CONTRIBUTING guide?

memsharded commented 3 months ago

Hi @tttapa

Thanks for your question.

To use a specific commit of the original repo, it is necessary to include or add that commit in the recipe that creates the package. What ConanCenter usually does when a repo doesn't publish releases, is adding a new cci.20240519 version based on the date, so it can get an updated package with latest changes, built on a specific commit. Such commit will be defined in the conandata.yml of the package recipe.

If you want to customize ConanCenter recipes to include later releases or other customizations that do not exist yet in ConanCenter (or might never be possible to be merged upstream), then you can easily have your own package, with the version you want, without conflicting with the existing ConanCenter ones. This is done by doing a fork of the conan-center-index Gtihub repo that contains all the recipes for ConanCenter, and then:

tttapa commented 3 months ago

I see, thanks for the quick response, the local recipes index feature looks promising!

If I understand correctly, this approach would require me to use the latest public Conan Center version in my project's conanfile, correct? I cannot set my custom version as the default requirement, otherwise users of my project will not be able to do the standard conan install . --build=missing without first conan createing or conan exporting the recipe from my custom conan-center-index fork, or without adding it as a remote.

Do you know of any plans to add direct Git support to Conan? E.g. by directly specifying a URL to the custom recipe, such as requires = "eigen/custom.20240519 @ git+https://github.com/my/conan-center-index-fork/.../eigen/conanfile.py" (where Conan would automatically download and export the custom recipe specified, using the custom version)? Has this been discussed elsewhere perhaps?

memsharded commented 3 months ago

If I understand correctly, this approach would require me to use the latest public Conan Center version in my project's conanfile, correct? I cannot set my custom version as the default requirement, otherwise users of my project will not be able to do the standard conan install . --build=missing without first conan createing or conan exporting the recipe from my custom conan-center-index fork, or without adding it as a remote.

No, you can use any version of ConanCenter, including older ones. The conan-center-index contains all versions information, and it can resolve to any of them, in the same way when using "local-recipes-index" and when using the actual ConanCenter. The only difference will be the existence of binaries. Note that if the fork of conan-center-index is aligned with the upstream, it might be even possible to use recipe from the local fork and a binary from the ConanCenter (for this, the recipe revisions must be identical, that is, the forked recipe must not have local changes)

Do you know of any plans to add direct Git support to Conan? E.g. by directly specifying a URL to the custom recipe, such as requires = "eigen/custom.20240519 @ git+https://github.com/my/conan-center-index-fork/.../eigen/conanfile.py" (where Conan would automatically download and export the custom recipe specified, using the custom version)? Has this been discussed elsewhere perhaps?

Yes, this has been discussed many times, and it is not planned. There are many limitations, complexities and problems associated with this approach. Conan follows the model of Maven for Java, with a strong abstraction over the dependencies references and using dedicated package servers, not coupling them with hardcoded source locations in recipes. The "local-recipes-index" feature was introduced to allow modifications to recipes in a convenient way, but for the most part is intended as a package creation flow, not as a package consumption flow, that is, packages should be created from the fork and stored in some package server.

tttapa commented 3 months ago

No, you can use any version of ConanCenter, including older ones.

Thanks, I meant that putting a Conan Center version (e.g. the latest, but could be any version) would be the only way to keep my project easily installable by users. IIUC, there are two options:

  1. Specify a public Conan Center version in my project's conanfile, e.g. requires = "eigen/3.4.0".
    In this case, a user can use my project after a simple git clone project; cd project; conan install . --build=missing, but they will end up with a build where some features are missing or broken, because they're missing some bug fixes to Eigen.
  2. Specify a version from my own Conan Center fork in my project's conanfile, e.g. requires = "eigen/custom.20240519".
    Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system, they will not be able to run conan install . --build=missing successfully out of the box.

Is there any way to make the installation process simpler for users of my project (i.e. no need to manually download and set up other repositories first), while also selecting a custom version by default so they don't end up with a subtly broken build?

memsharded commented 3 months ago

Thanks, I meant that putting a Conan Center version (e.g. the latest, but could be any version) would be the only way to keep my project easily installable by users. IIUC, there are two options:

It depends on who your users are. There are wildly different use cases, if those are other colleague developers in the same organization or company, or if you are planning to release something as open source library.

Specify a public Conan Center version in my project's conanfile, e.g. requires = "eigen/3.4.0". In this case, a user can use my project after a simple git clone project; cd project; conan install . --build=missing, but they will end up with a build where some features are missing or broken, because they're missing some bug fixes to Eigen.

Yes, and they can also have other bugs when using a non-released version of eigen, because the maintainers didn't put the typical pre-release effort to stabilize and do more comprehensive tests. This is not an issue of the package manage per-se, but of the library and its maintainers. My experience suggests that if Eigen had very critical updates that haven't been released and the users wanted to use, we would already have an updated eigen/cci.date. This is something that can be discussed, we are not closed to have these versions in ConanCenter, if it makes sense and it is good for the wider community, it might be consider to create a cci.date release, that is quite simple.

Specify a version from my own Conan Center fork in my project's conanfile, e.g. requires = "eigen/custom.20240519". Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system, they will not be able to run conan install . --build=missing successfully out of the box.

That is the thing that depend on who your users are. For organizations, this is not an issue, because they basically get the configuration from a conan config install command, getting all the organization custom profiles, settings, remotes definitions, plugins and many more, and packages like eigen/custom.20240519 would already be in your own package server, and in cases it is a library in pre-compiled format for the different binary configurations, saving lots of time of CI and developers.

Now I can be sure that users have a working version of Eigen, but unfortunately, this significantly raises the barrier to entry for users, making it harder for them to install and use my project. Because unless they first go and download my Conan Center fork and add it as a local remote on their system

And if your users are open-source users, is it really necessary that they use the very latest eigen from source? What actual crashes or issues happen if they don't? Maybe it is worth to consider a cci.date version in ConanCenter?

In any case, "significantly raises the barrier" means:

$ git clone <url-to-myrepo>
$ conan remote add local myrepo

If it is that important to use customized versions that cannot exist in ConanCenter, then those are relatively straightforward setup instructions, specially for open-source users that are typically faced with much more complicated project setups.

tttapa commented 3 months ago

Thanks, I appreciate the advice.

One more question on this topic:
Do you have any pointers for continuous integration? What would be the best way to force Conan to install the very latest commit on the main/master branch of (one of) my dependencies?

Would your recommendation be different for dependencies that I maintain myself (i.e. where I could do git clone my-dep; conan export ./my-dep), versus third-party dependencies that come from the CCI?

memsharded commented 3 months ago

Do you have any pointers for continuous integration? What would be the best way to force Conan to install the very latest commit on the main/master branch of (one of) my dependencies?

When using a package manager, the natural flow is enabling CI to build a new package version (or revision) for the commit or tag you want to make available for others to use. For first-party code, the recommendation would be to put the conanfile.py in the source repo of the code that it is packaging, for several reasons:

The issue is that we haven't managed yet to write comprehensive CI docs, the plan is it that it will be added to https://docs.conan.io/2/devops.html section, but it is difficult to get the time. Conan 2 has some important features to be able to support CI at scale, like the conan graph build-order command or the lockfiles to be able to distribute builds for multiple binary configurations, without needing to replicate the dependency graph topology on the CI system. I know it is not as nice as docs, but the tests in https://github.com/conan-io/conan/blob/develop2/test/integration/lockfile/test_ci.py should be understandable, and would be a very good reference of the main CI ideas. Note that these CI tests are not for the specifics of building a specific package when it gets some code changes, but for the most challenging CI process that is "now that I have a new package version/revision, what other packages need to be rebuilt in which order to validate that my main products are not broken by this new package".

The plan is to add this documentation in the following months.