search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
948 stars 1.73k forks source link

[question] [docs] Rules about upgrading the requirements #2477

Open prince-chrismc opened 4 years ago

prince-chrismc commented 4 years ago

Given that a recipe has many dependencies, it's very common that one of it's requirements can have a higher version available in CCI then whats written in the recipe.

For header only there is no issue as the requirement can be overridden by the consuming conanfile, however compiled libs may suffer from API/ABI incompatibilities. This makes resolving conflicts very painful when two graphs overlap. Typically we get stuck building the package locally.

Are we permitted to open PR's where the only change it to the version of a required dependency?

Ideally as a consumer, it's preferable to pull everything from CCI rather than building things locally.

jgsogo commented 4 years ago

Yes, the main problem is when there is a conflict because two graphs require a different version of some requirements. We would really want to avoid this situation, so it is perfectly ok to open a PR to upgrade only a requirement.

What else can be done?

Any other alternative?

SSE4 commented 4 years ago

let's try to illustrate the situation on the hypothetical example. for instance, we have a library boost, which did release 1.72.0. at the time of release, boost had a dependency on another library, named zlib, and the latest release of zlib was 1.2.11. it means in conan we would have a relationship like: boost/1.72.0 -> zlib/1.2.11. okay, we have conan packages for both boost and zlib. now imagine that zlib suddenly made a new release, e.g. 1.2.12. should we upgrade boost release 1.72.0 right away to reference new zlib release? it might look that obvious decision is to upgrade, but is it really so simple? when boost maintainers did 1.72.0 release they run their regression tests with zlib 1.2.11, they tested various configurations (different compiler versions, operations systems, etc.) to ensure stability. in conan we don't run so complete testing matrix, we only ensure that packages successfully build and pass simple test package. we never run test suites, and we never run the same set of supported configurations. the same probably applies to every package. I understand zlib example might be not so illustrative, as it's pretty stable and has a very stable interface, but in general - does it mean we always have to keep these references on bleeding edge to ensure there are no conflicts?

uilianries commented 4 years ago

Most of projects don't use a specific version for dependencies, it's hard for us tracking it. Indeed should reasonable keeping them aligned due the compatibility, but would need to check which version the author used, the latest version available when that release was made (by date)?

SSE4 commented 4 years ago

@uilianries yes, there a projects like that. on the opposite side, there are cases then versions of dependencies are very hard requirements. for instance, your favorite protobuf - in case the project includes sources generated with protoc, it will need to link with the same version of protobuf, as generated sources contain hard version check.

prince-chrismc commented 4 years ago

My particular pain is fmt, which has a very aggressive release cycle, much more than it's dependents. Juggling 4 projects which require it is a growing pain. Version 6.0, 6.2, and 7.0 are requirements but all support 7.0 since they are not impacted by the API changes.

I would have to agree that automatically upgrading the requirements would be a sticky situation. To add to SSE$'s points, some recipes require a different version depending on the version (but these could be changed).

I have to test my code base when I changing dependencies, I may submit a few PRs after I am confident there's no issue. There will have to be a case by case review.

Would it be possible for the bot to fill an issue that shows all the conflicts?

SpaceIm commented 4 years ago

Auto updating recipes doesn't scale. What about recipes with a lot of dependencies (and big binaries)? Look at GDAL. Do you really want to create another revision each time one of its many dependencies release a new version? I think that CCI recipes should support version range, and eventually a "preferred" version if reproducibility is an issue (but a lock file should be what ensure reproducibility, not specific hardcoded version in recipes). Consumers lose a lot of information without version range, they have to figure out themself compatible versions, while it is basically what a package manager should provide.

timblechmann commented 3 years ago

looking at this from a different angle: as i've came across this when trying to resolve dependency conflicts in my project. i certainly understand the combinatoric explosion of configurations and problems that arise from dependency management, but i tend to run into issues where different dependencies pull in different versions of other dependencies.

libcurl is a great example:

11:35 $ git grep libcurl\/
recipes/cfitsio/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/cpr/all/conanfile.py:        self.requires("libcurl/{}".format("7.67.0" if not self._supports_openssl else "7.69.1"))
recipes/czmq/all/conanfile.py:            self.requires("libcurl/7.71.1")
recipes/date/all/conanfile.py:            self.requires("libcurl/7.69.1")
recipes/gdal/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/kcov/all/conanfile.py:                "libcurl/7.64.1",
recipes/libgit2/0.27.x/conanfile.py:            self.requires("libcurl/7.67.0")
recipes/libmediainfo/all/conanfile.py:        "libcurl/7.69.1",
recipes/mariadb-connector-c/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/poppler/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/proj/all/conanfile.py:            self.requires("libcurl/7.73.0")
recipes/sentry-native/all/conanfile.py:            self.requires("libcurl/7.71.0")
recipes/tgbot/all/conanfile.py:        self.requires("libcurl/7.72.0")

different recipes depend on different libcurl versions. typically it seems to be the "most recent version when writing a packge".

openssl is another, where recipes tend to depend on 1.1.1d, 1.1.1g, 1.1.1h and 1.1.1i. in the case of openssl where 1.1 has a stable ABI, but 'letter' releases tend to be security fixes, it seems to be rather important to actually pull in the "latest 1.1" rather than the "latest at the time of writing the recipe".

this makes me wonder if there should be some kind of recommendation to require with version ranges rather than a specific version? in the world of package managers like debian packages this seems to be quite common

prince-chrismc commented 3 years ago

The challenge is the ABI correctness is not always guaranteed. with libcurl https://abi-laboratory.pro/index.php?view=timeline&l=curl every minor release is an ABI break, so your range would be >=7.73.0 && <7.74.0 but then you tradeoff on reproducibility, and you would be generating different package_id... you would still need to solve the combinatric hell that we already live with =D

timblechmann commented 3 years ago

with libcurl https://abi-laboratory.pro/index.php?view=timeline&l=curl every minor release is an ABI break

this is certainly not true: if you consult the "Backward Compat.", the only change ABI break listed here was 7.62.0 where the value of an enum was changed. all other changes are backwards compatible. of course ABIs can break, but in these cases one would have to decide on case-by-case basis. considering that a package could be built from sources, one may not even worry about ABI compatibility, but merely about API compatibility.

then you tradeoff on reproducibility

well, conan already has issues when it comes to reproducibility: since recipes aren't versioned one build of libcurl=7.73.0 might be quite different than another version of libcurl=7.73.0 😬

out of curiosity: in what way is conan different than debian packages? after all version ranges work quite well for debian, which is a rather mature project with a huge number of packages

prince-chrismc commented 3 years ago

There's a reason the squares are colored... orange, yellow, green have meaning.

You might find backwards compatible with a sprinkle of surprise behavior ! https://abi-laboratory.pro/index.php?view=compat_report&l=curl&v1=7.69.1&v2=7.70.0&obj=19c9c&kind=abi#c_1

last != last ... Oops!

since recipes aren't versioned

They are, CCI fully supports revisions check https://conan.io/center/libcurl you need to enable you local client for this feature to be enabled. Check this blog out https://blog.conan.io/2019/03/12/New-conan-release-1-13.html

timblechmann commented 3 years ago

You might find backwards compatible with a sprinkle of surprise behavior ! https://abi-laboratory.pro/index.php?view=compat_report&l=curl&v1=7.69.1&v2=7.70.0&obj=19c9c&kind=abi#c_1

reading this it is a little paranoid:

since recipes aren't versioned

They are, CCI fully supports revisions check https://conan.io/center/libcurl you need to enable you local client for this feature to be enabled. Check this blog out https://blog.conan.io/2019/03/12/New-conan-release-1-13.html

interesting. i need to have a look how to integrate this into my project. though i wonder why this is per-local-client rather than per-project 🤔

though again: in which way are conan recipes different to debian packages?