[question] pre install hook

[nicosmd]

What is your question?

Hey,

I'm currently trying to implement a hook in order to warn if users haven't installed the latest config version of our config repository based on a version file within the config repo. Therefore I would need to hook-in as early as possible and before the configs becoming relevant. Hence, I think, this hook should take action before installing any package but I couldn't find any hook which fits this requirement. Is there a reason it is not possible to create hooks before install and could you recommend which hook would be the best for this use-case? Or is there even maybe a smarter way to achieve this goal?

Thank you very much!

Have you read the CONTRIBUTING guide?

• [X] I've read the CONTRIBUTING guide

[memsharded]

Hi @nicosmd

I'm currently trying to implement a hook in order to warn if users haven't installed the latest config version of our config repository based on a version file within the config repo.

This looks like a chicken-and-egg problem? The hook itself comes from a config. You need to have an updated hook to validate the config, so need to update the config first in order to be able to do that?

Therefore I would need to hook-in as early as possible and before the configs becoming relevant. Hence, I think, this hook should take action before installing any package but I couldn't find any hook which fits this requirement. Is there a reason it is not possible to create hooks before install and could you recommend which hook would be the best for this use-case? Or is there even maybe a smarter way to achieve this goal?

Hooks are designed for pre-post recipe methods extensions, but not for general tasks. In Conan 2.0, those would be plugins as the profile.py plugin or the compatibility.py plugin.

Still, I don't see how a plugin can be used to force a certain version of the config, because the plugin itself is also part of the config.

[nicosmd]

Hey @memsharded thank you very much, I will have a look inside compatibility and profile plugins. I would do it like this:

1. check version of the current installed config using a version file which is part of the config repo
2. performing a http request to bitbucket to get the version of the latest approved config version or using git tags to fetch all available versions
3. warn the user that a newer version would be available and provide the command needed to update the config or maybe ask to automatically install the newest config and then exit so that the user would be forced to redo the command with the new config

this wouldn't be a chicken-egg issue or? I mean sure, the user needs to at least install the config manually once to activate the hook/plugin, which is fine. But from then it should be possible to perform this check, or am I missing something?

[memsharded]

Thanks for the explanation, it now makes more sense.

I will have a look inside compatibility and profile plugins.

The compatibility.py is bad for this, while the profile one, while doable, might be a bit of too much. The problem with the profile.py plugin is that it will be invoked twice for every command that needs profiles, one for the "host" profile and one of the "build" profile.

There would be also a problem of executing this once Conan is already running: the configuration has already been loaded, all the commands, plugins, deployers, custom settings, etc. Feching a new config will not automatically update the current running instance configuration. This means that the command should update, then abort, then require the user to re-run.

Isn't it a concern that this will slow down every user conan xxx command doing such a remote http request each time? Especially using git tags to fetch all versions can be very slow. What happens for offline work?

In Conan 1.X we had the capability of calling periodically and automatically a memorized conan config install. It was removed because of being fragile and problematic.

Is the pain that big? Are developers that often outdated?

I am working on a new capability, using config from Conan packages. This will allow conan config install "myconfig/[>=1 <2]" --type=pkg (or just myconfig/[*] for the latest always), for example, and that will be able to automatically use the latest config in the desired range. Maybe this would be good enough to mitigate the pain?

[nicosmd]

Thank you very much for such an elaborate reasoning of my idea. This helps me a lot.

I think it would be not that worse using the profile plugin even if the check performs twice. When using one http request to fetch the latest config version it would be quite fast and in most cases you wouldn't recognize it. I also think that we do not perform conan commands very frequently when developing locally. Our recommended flow is to use conan install once and then continue using plane cmake commands. So it would be executed at the right time when I need to fetch newer packages for example, where a lot network communcation is performed anyway. I can also imagine to make the check deactivable using an environment variables for example so it is not done during CI/CD builds as there we always fetch the right version. For working offline it would be fine to skip the check, when the request fails. It should be just a lightweight helper to get informed about config updates and not a hard requirement to be performed. So my feeling is that it would be ok. Also re-run the conan command when a new config was installed does not feel like a blocker as it does not happen frequently and we are familiar with similar patterns used by git pre-commit hooks for formatting.

Is the pain that big? Are developers that often outdated?

I wouldn't say it is a big pain point but it will definitely save a bit of debugging time and reduce frustrations as this is basically the number one troubleshooting step to do, followed by not authenticated to the conan remote, and when there would be some automation to ensure this, it would make it much easier and less critical to introduce configs changes which are breaking. With breaking change I mean that we have quite often the scenario, that when introducing a new dependency to the graph we often need to set immediately some options for it to make it working. Then the next days it is foreseeable that we are getting requests by developers, why latest changes does not build locally anymore. So I think when it would be a low hanging fruit to introduce at least a warning, it would worth it. When it is is too complicated I would live with it.

The new feature to package the configs as a conan package sounds very interesting but would that solve the issue? From my understanding I would still need to update my cache from time to time to get latest configs. Otherwise, it will be used what is in my cache when it fits the version range, or? Maybe I don't yet understand all consequences, when the config would be a package itself.

[memsharded]

I think it would be not that worse using the profile plugin even if the check performs twice. When using one http request to fetch the latest config version it would be quite fast and in most cases you wouldn't recognize it. I also think that we do not perform conan commands very frequently when developing locally. Our recommended flow is to use conan install once and then continue using plane cmake commands. So it would be executed at the right time when I need to fetch newer packages for example, where a lot network communcation is performed anyway. I can also imagine to make the check deactivable using an environment variables for example so it is not done during CI/CD builds as there we always fetch the right version. For working offline it would be fine to skip the check, when the request fails. It should be just a lightweight helper to get informed about config updates and not a hard requirement to be performed. So my feeling is that it would be ok. Also re-run the conan command when a new config was installed does not feel like a blocker as it does not happen frequently and we are familiar with similar patterns used by git pre-commit hooks for formatting.

Maybe another approach to consider is to overwrite with your own custom commands some of the Conan commands. Then implement the functionality of the config first thing you do in your command. I am not saying it would be perfect, or even better, just another idea.

The new feature to package the configs as a conan package sounds very interesting but would that solve the issue? From my understanding I would still need to update my cache from time to time to get latest configs. Otherwise, it will be used what is in my cache when it fits the version range, or? Maybe I don't yet understand all consequences, when the config would be a package itself.

One of the challenges of regular conan config install is to use the right url, with the right "last version". That requires some effort from the developers side.

If the command is easy to remember, and no effort to know what is the latest, but it is automatic, it helps a lot to reduce the mental overhead, and it is more likely to be incorporated in developer flows, in the same way more or less that we often do apt-get update before an apt-get install, just in case to update the repos if we hadn't in some time.

Thanks very much for the feedback. I think it is still a bit early to consider a very early stage plugin, like conan_init.py or something like that that executes on every Conan command invocation, but I will keep your use case in mind. Please let me know how it goes with your profile.py attempt.

[nicosmd]

Maybe another approach to consider is to overwrite with your own custom commands some of the Conan commands. Then implement the functionality of the config first thing you do in your command. I am not saying it would be perfect, or even better, just another idea.

That sound also very reasonable but I would like to avoid replacing default conan commands by custom overloaded commands. We would like to keep the workflow as default as possible to allow every conan familiar developer to feel comfortable with our build system.

I've just tried out to implement it using profiles plugin and it works like a charm and feels quite good. I used the Bitbucket API to fetch the latest version file and set a timeout to 500ms. If it does not response in this time it is considered to not work and the build just continues. I have implemented a user choice if the config shell be updated automatically and abort or not.

I was even able to mitigate the issue of evaluating the plugin twice by adding:

evaluation_count = 0

def profile_plugin(profile):
   global evaluation_count
   evaluation_count += 1
   if evaluation_count == 1:
       _check_latest_config_installed()

to my own surprise, this actually works. Sure a dedicated plugin for this would be neat but it works and provides what we need.

Thank you very much for your help!

[memsharded]

evaluation_count = 0

Sure, globals FTW 😂 Not beautiful, but indeed does the job

Thank you very much for your help!

No prob, love to help, happy that it works!

Sure a dedicated plugin for this would be neat but it works and provides what we need.

I am labeling this as "look into" to discuss with the team, and consider a dedicated plugin. Most likely a bit too early and will not be a priority, but just in case. Thanks for the feedback!

[nicosmd]

That sounds great!

I'm still wondering how others solves this issue because my feeling is that when using conan in a larger project you came across the question how to manage the config version across the project and the developers at some point. I haven't seen many question in this direction yet and my solution obviously feels a bit like solving a special use case. Are others bascially doing well with encourage developers to do a conan config install frequently like when using apt, as you have mentioned? Do you maybe have rough overview about how this is solved usually?

[memsharded]

I'm still wondering how others solves this issue because my feeling is that when using conan in a larger project you came across the question how to manage the config version across the project and the developers at some point.

No, you are right, this concern has also been raised before, and this is why Conan 1.X attempted to run automatically conan config install scheduled every X time. We decided to remove it because of the many issues, but that doesn't mean that we know the concern is still there.

We haven't prioritized it again in 2.0 yet because in practice we have seen that configuration doesn't change that often, in more or less normal flows it can remain unchanged for weeks without issues. And in less stable periods it is not that developers need to change it every hour, so just a notification to the teams that a new config is there is mostly enough. In majority of situations for developers, the changes to the configs are not breaking, the most common cases is adding new profiles for some new configuration or platform to support, so not critical to update immediately. So breaking changes that force a quick organization update of configs are infrequent in practice. So it is not perfect or optimal, but a relatively pragmatic approach.

[memsharded]

Hi @nicosmd

The conan config install-pkg "myconfig/[>=1.0 <2.0]", that is, the possibility of using Conan packages for configurations was released in Conan 2.2, like 2 months ago. Have you had the opportunity to give it a try? The idea is that it simplifies a lot the developers flow to be updated, as they don't even have to use a large URL, it allow version ranges ([*] just for latest) while also increasing a lot the reproducibility, because the config package version can also be locked in lockfiles.

[nicosmd]

Hey @memsharded

I have tested it a little and I think it is a great feature. Especially that it allows version ranges has very interesting use cases from DevOps perspective. Besides that it is a really neat feature that the config can now be locked as well!

What I'm still missing is the possibility to set the config version required by a package in a file stored in the repository (maybe .conanrc) which will inform the user that the currently installed config does not match the required config by the repo or maybe even install the required config if an alternative CONAN_HOME is specified in the .conanrc file.

[memsharded]

What I'm still missing is the possibility to set the config version required by a package in a file stored in the repository (maybe .conanrc) which will inform the user that the currently installed config does not match the required config by the repo or maybe even install the required config if an alternative CONAN_HOME is specified in the .conanrc file.

I see, but this would be a different story than the one I first understood, which was mostly about just downloading the latest available config in the server if the one local is not enough. But from what you say here is not about the latest, but about the capacity to install any specific configuration for different workspaces/.conanrc?

[nicosmd]

Either to install a specific configuration for different workspaces or to warn if the globally installed config version does not match the required version by a workspace. But the download issue was definitely also part of my initial problem and I think it will solve some issues we had.

[memsharded]

I think the original issue has changed, and we are no longer talking about a pre-install-hook, but mechanisms to keep the config updated or validated against different conditions, which might be done by different approaches. One of them would be using workspace feature to allow at least validation of the installed config version.

We need to think a bit more about these use cases.