search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
7.95k stars 951 forks source link

[question] why does recipe revision change when building recipe from conan center index? #16510

Closed wuziq closed 1 week ago

wuziq commented 1 week ago

What is your question?

TLDR: The conandata.yml changes depending on whether the recipe is built directly vs built via --build=missing. Why?

Details:

We are building zlib/1.3.1:

conan create --build=missing --name zlib --version 1.3.1 -pr:a=default -r local-index ./conan-center-index/recipes/zlib/all

This yields:

...
======== Exporting recipe to the cache ========
zlib/1.3.1: Exporting package recipe: /__w/2/s/conan-center-index/recipes/zlib/all/conanfile.py
zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it...
zlib/1.3.1: Calling export_sources()
zlib/1.3.1: Copied 1 '.yml' file: conandata.yml
zlib/1.3.1: Copied 1 '.py' file: conanfile.py
zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch
zlib/1.3.1: Exported to cache folder: /root/.conan2/p/zlib95420566fc0dd/e
zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-06-19 20:49:50 UTC)
...

However, when we build another package that depends on zlib, the recipe revision is different. For example, with boost:

conan create --build=missing --name boost --version 1.71.0 -pr:a=default -r local-index ./conan-center-index/recipes/boost/all

This yields:

...
======== Computing dependency graph ========
zlib/1.3.1: Not found in local cache, looking in remotes...
zlib/1.3.1: Checking remote: local-index
zlib/1.3.1: Downloaded recipe revision f52e03ae3d251dec704634230cd806a2
...

Comparing the recipe revisions e20 and f52, the only difference is in the conandata.yml file. When we build zlib directly, the conandata.yml matches exactly what is contained in the conan-center-index. However, when zlib is built via the boost build, zlib's conandata.yml is much smaller, containing only the source and patch paths for version 1.3.1.

I found this thread: https://github.com/conan-io/conan/issues/13874 It mentions hook_reduce_conandata.yml from https://github.com/conan-io/hooks.

Questions:

We don't have this hook installed, so why are we seeing this behavior while building boost?

Maybe it happens automatically for packages that are built because of --build=missing? If so, we should install this hook so that we get the same behavior when building packages directly, correct?

If so, what's the correct way to install this hook in Conan2? (the hooks readme is geared towards Conan1 - there is no conan config set in Conan2).

Have you read the CONTRIBUTING guide?

AbrilRBS commented 1 week ago

Hi @wuziq thanks for the question. You're mostly on point with your discoveries, and it does boil down to the trimming of the conandata.yml file!

We don't have this hook installed, so why are we seeing this behavior while building boost?

As I see, you're using the local recipes index remote type, and one of its goals is to match the behaviour of Conan Center Index, where we do trim the conandata for each version, so that recipes fetched from this remote type algo get trimmed. You're not seeing this behaviour when compiling it because the trim happens internally within the local remote.

For you to get the same behaviour of locally created packages, you do need to have a post_export() hook that calls [conan.tools.files.conandata.trim_conandata()](https://docs.conan.io/2/reference/tools/files/basic.html#conan.tools.files.conandata.trim_conandata). This file needs to go into your <conan2_home>/extenions/hooks/ folder (docs for naming conventions etc here), something like:

_~/.conan2/extensions/hooks/hook_trimconandata.py

from conan.tools.files.conandata import trim_conandata

def post_export(conanfile):
    trim_conandata(conanfile)

The way to share/install this across your developers would be using the conan config install command and associated features, docs here

Let me know if this helps :)

wuziq commented 1 week ago

Fascinating, thank you! This leads me to another question. When building our own private conan center index, should we not be calling conan create as I showed above? Should we be building the recipes differently? We don't mind using the hook, but we're curious whether we can avoid adding the hook.

We are following this: https://docs.conan.io/2/devops/conancenter/hosting_binaries.html It says to create a list of the packages we want, and to then call conan create --build=missing.

However, I just noticed in another document (https://docs.conan.io/2/devops/devops_local_recipes_index.html#building-binaries-from-a-private-conan-center-index-fork), it says to use conan install --requires=....

Which is correct?

Maybe we are not calling conan create correctly?

AbrilRBS commented 1 week ago

Extra note about my prior message: Note that trim_conandata() raises if any of your packages does not have a conandata.yml file. If this is undesired, pass False as the second parameter which will disable the behaviour and silently do nothing

When building our own private conan center index, should we not be calling conan create as I showed above?

You are calling conan create mostly correctly :) I'll explain the differences between the install and create below, but for now:

The docs mention conan install because the idea there is that you're not creating the packages, but using them as part of the dependencies for a project you're running, but which you're not creating the package at that moment. Your conan create call is best suited for your case, as you want to create the zlib package and be able to later upload it to a remote/use it from your cache, which is different than having the recipe be available from a local recipes index :)

wuziq commented 1 week ago

Thank you!