[suggestion] Allow specifying commit hash when installing config from git repository with `conan config install`

[Corristo]

What is your suggestion?

Hello,

the project I work on keeps the conan config files in the project's git repo, with a post-checkout hook that calls conan config install to ensure that the correct config files for that specific commit are installed. We recently modularized our code a bit and moved parts of it to a different repo, and we have duplicated the conan config files there for now. To avoid issues when the config is accidentally only changed in one of the repos we'd like to move the conan config files to a dedicated repo so that they can be shared properly.

However, conan config install only allows to specify additional arguments for the git clone call, and since git clone doesn't allow to specify an individual commit we can only install from a branch or tag. This makes it inconvenient to pin the exact version of the config files in our source code repos: Whenever someone makes a change to the config files in the config repo we need to add a new tag (or branch), so that we can explicitly check out this commit when calling conan config install in our post-checkout hook.

If we were to always install the config from the main branch instead of a pinned commit there might be issues that older commits can no longer be rebuilt if we make some incompatible changes to our conan config, which also isn't ideal

It would be very helpful if conan config install were to gain a command-line flag to specify the hash of a commit in the config repo, which it will automatically switch to after cloning and then install the config from there.

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @Corristo

Thanks for your suggestion.

The conan config install has an extra --args argument that forwards arguments to the underlying git clone, so this is possible:

$ conan config install <url> --args "-b my_branch"

to be able to use the specific branch you want.

Also, if you are concerned about reproducibility, then maybe you are interested in checking the new conan config install-pkg command, that uses Conan packages to store the configuration: https://docs.conan.io/2/reference/commands/config.html#conan-config-install-pkg

This allows some interesting things:

• using version-ranges for automatic updates
• Storing the exact version and revision of the configuration in lockfiles for future reproducibility
• Defining a specific effect on the package_id of packages built under this configuration
• Using Conan recipes power to create and manage the configurations

[Corristo]

Thanks for your reply.

I am aware of the --args option to specify a branch or tag, but I'd really like to be able to specify a commit by its hash. If someone were to check out a commit from today in 5 years time I want them to get the same conan config that I get today. If I only specify a branch that branch might have been updated by then, and they don't get the same config. One way to work around that issue is to create a tag in the config repo whenever the configuration is updated, because git clone's -b also accepts tags. But doing that likely will lead to almost as many tags as there are commits, and if you really want to tag certain commits (e.g. to tag the config version used for a particular release of the product) these tags will be hard to find among all the other tags.

My current workaround is to have the post-checkout hook clone the config repo, then perform the checkout of the correct commit, and then call conan config install with the path to the cloned repo instead of using the git repo directly in the conan config install call. So this feature is definitely only a quality-of-life improvement, but I assumed that more people might be interested in being able to reproducibly install the conan config, which is why I opened this issue.

I'll have to look into conan config install-pkg. One downside is that this obviously requires the remote to be set up correctly already, and currently the remotes.json is also part of the configuration we install. But it is probably a good idea to keep that somewhere else anyway, as the remote might change in the future. Will conan config install-pkg ensure that the configuration is exactly as described by the package, i.e. will it delete hooks, extensions, profiles, etc that are in the config directory but not the config package? Because that is another pain point I currently have with conan config install.

[memsharded]

I am aware of the --args option to specify a branch or tag, but I'd really like to be able to specify a commit by its hash. If someone were to check out a commit from today in 5 years time I want them to get the same conan config that I get today. If I only specify a branch that branch might have been updated by then, and they don't get the same config. One way to work around that issue is to create a tag in the config repo whenever the configuration is updated, because git clone's -b also accepts tags. But doing that likely will lead to almost as many tags as there are commits, and if you really want to tag certain commits (e.g. to tag the config version used for a particular release of the product) these tags will be hard to find among all the other tags.

I see, yes, it is true that it is not possible to specify commits, only tags. So far, configuration repos have been used with tags or branches, nobody requested a way to checkout commits that are not tagged. Note that it is also possible to achieve the same result with something like:

conan config install https://github.com/<orgname>/<reponame>/archive/<commit>.zip

At least in Github, but I think other servers might have similar functionality.

Will conan config install-pkg ensure that the configuration is exactly as described by the package, i.e. will it delete hooks, extensions, profiles, etc that are in the config directory but not the config package? Because that is another pain point I currently have with conan config install.

No, the behavior is the same as conan config install, it is additive, aggregating and overwriting files with the same name, but not removing things. That should be done by removing the Conan cache home, typically just rm -rf ~/.conan2, or starting from a new Conan home (with the CONAN_HOME env-var). I think there is some ticket opened for implementing some removal capabilities in conan config install, but I can't find it now. Anyway, this would be a different topic to discuss in a separate issue, better keep this one focused.

[memsharded]

Hi @Corristo

Any feedback? Is the conan config install https://github.com/<orgname>/<reponame>/archive/<commit>.zip approach good for your use case?

[Corristo]

Technically our git server supports downloading zip files, but because the repo isn't public it asks for username and password when trying to download the zip file, and thus I can't automatically install it as part of a post-checkout / post-merge hook. I've therefore stuck with my own approach of manually cloning the repo to a temporary directory, doing the checkout of the correct commit and then installing from that directory.

I had also tried to use conan config install-pkg, but that requires the remote to be set up correctly by the user, whereas with my I approach I can automatically set up the correct remote (and remove the conancenter one) for the other developers.

I still think that allowing to specify the commit hash is a nice-to-have feature, but I can also understand if you'd rather spend your time on something else and close this issue.

[memsharded]

I still think that allowing to specify the commit hash is a nice-to-have feature, but I can also understand if you'd rather spend your time on something else and close this issue.

Yes, so far, with the information we have, it is a bit difficult to prioritize it, unfortunately our backlog is simply overflown.

Maybe you want to give it a try and try to contribute the feature? I think it shouldn't be very challenging.

[Corristo]

I'm not that familiar with python, but as far as I can tell it should be pretty straight forward.

Is there a naming convention for command-line arguments like this one that only apply under certain conditions that I should follow when naming the option? When someone tries to install the config from a local directory or a downloaded archive, but specifies the new option, should that be a warning ("Warning: \<argument> is ignored") or a hard error? Are the any constraints with respect to API compatibility of the Python API that I need to be aware of?

[memsharded]

Is there a naming convention for command-line arguments like this one that only apply under certain conditions that I should follow when naming the option?

Not really. This is a not very clean scenario of an option that applies only to some particular cases, not making sense for others, but similar to the --verify-ssl that only applies to downloading tarballs, but not other origins for config install. So something like --git-commit would be explicit.

When someone tries to install the config from a local directory or a downloaded archive, but specifies the new option, should that be a warning ("Warning: is ignored") or a hard error? Are the any constraints with respect to API compatibility of the Python API that I need to be aware of?

We don't necessarily do "defensive" programming and testing for all potential cases. If the argument is called --git-commit and users use it for a regular zip tarball, a warning is nice, but not that critical (warnings are often disregarded anyway). A hard error would be correct, we sometimes add them, but it is also not critical if devs do not add and test every potential cross-interaction with every other argument.