[service] Removing `compiler` from `package_id` causes libstdc++ compatibility errors in tool_requires packages

[samuel-emrys]

What is your problem/feature request?

As per https://github.com/conan-io/conan-docker-tools/issues/501, removing compiler from package_id with a statement similar to the following is the incorrect model to model package requirements for tool_requires packages.

def package_id(self):
   del self.info.settings.compiler

The compatibility model that should be used here is that any compiler newer than the version used to build the library is compatible. The reason for this is that building an executable with gcc 11 will use GLIBCXX_3.4.30, and so some assumptions are injected into the executable about which libstdc++ library version is available in the environment. If this executable is used in an environment where gcc 10 is used, the installed libstdc++ library will only have symbols compatible with up to GLIBCXX_3.4.28 and the executable will fail to execute.

One example of this in CCI currently is the doxygen recipe. Because conan 2 is building packages using gcc 11, attempting to use the doxygen recipe using the conanio/gcc10-ubuntu16.04 docker image fails with errors similar to the following:

/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
CMake Warning at .conan/p/cmake5b7ee79318ee9/p/share/cmake-3.25/Modules/FindDoxygen.cmake:492 (message):
  Unable to determine doxygen version: 1
Call Stack (most recent call first):
  .conan/p/cmake5b7ee79318ee9/p/share/cmake-3.25/Modules/FindDoxygen.cmake:655 (_Doxygen_find_doxygen)
  docs/CMakeLists.txt:1 (find_package)
-- Found Doxygen: /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen  found components: doxygen dot 
/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
/builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen: /usr/local/lib64/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by /builds/user/project/.conan/p/doxyga56639b015d56/p/bin/doxygen)
CMake Error at .conan/p/cmake5b7ee79318ee9/p/share/cmake-3.25/Modules/FindDoxygen.cmake:734 (message):
  Unable to generate Doxyfile template: 1
Call Stack (most recent call first):
  docs/CMakeLists.txt:1 (find_package)
-- Configuring incomplete, errors occurred!
See also "/builds/user/project/build/Debug/CMakeFiles/CMakeOutput.log".
*********************************************************
Recipe 'conanfile.py (project/0.1.0)' cannot build its binary
It is possible that this recipe is not Conan 2.0 ready
If the recipe comes from ConanCenter check: https://conan.io/cci-v2.html
If it is your recipe, check if it is updated to 2.0
*********************************************************
ERROR: conanfile.py (project/0.1.0): Error in build() method, line 130
    cmake.configure()
    ConanException: Error 1 while executing

To fix this, there are a few options:

1. Always build packages that delete compiler from package id with a very old compiler. This will just hide the underlying issue but it will give broad compatibility quickly. Could be a good short term solution.
2. Introduce a better way to model libcxx versions and their compatibilities. This is a harder problem to tackle as it would likely be a breaking change but is probably the most desirable long term solution.
3. Prohibit the practice of removing compiler from package_id in favour of more robust compatibility modelling using the compatibility() function. This is the best short/medium term solution and could be akin to (untested):

def compatibility(self):
    # is there a better way of reading all possible versions from settings.yml?
    settings_yml_path = pathlib.Path(os.environ.get("CONAN_HOME", pathlib.Path(pathlib.Path.home(), ".conan2")), "settings.yml")
    with open(settings_yml_path, "r") as f:
        settings_yml = yaml.safe_load(f)

    return [{"settings": [("compiler.version", v)]}
        for v in settings_yml["compiler"][str(self.settings.compiler)]["version"] if v <= Version(self.settings.compiler.version)]

Specifically, the documentation has the following suggestion:

Recipes that primarily provide compiled applications (e.g. b2, cmake, make, ...), which typically applies to packages that are consumed as tool requires) must list all the settings as well, as they are required during package creation. However, it is advised that the compiler setting is removed one in the package_id() method as follows:

def package_id(self):
   del self.info.settings.compiler

This reflects those cases where tools are consumed exclusively as executables, irrespective of how they were built. Additionally, this reduces the number of configurations generated by CI.

I'm proposing this should be removed and should be considered grounds for rejection of a PR in a review in favour of some agreed upon compatibility() function definition.

Another conversation that may be worth having is whether there's a better way to model these requirements explicitly rather than via compiler version. This isn't a great metric for evaluating which clang compiler versions can be used, for example, given it can also build using libstdc++ and is not just constrained to libc++. This is somewhat related to https://github.com/conan-io/conan/issues/3972

[jcar87]

Hi @samuel-emrys - thanks for opening this issue and your analysis.

The approach followed by Conan Center is roughly what you list here:

Always build packages that delete compiler from package id with a very old compiler. This will just hide the underlying issue but it will give broad compatibility quickly. Could be a good short term solution.

That is, we try and ensure that packages that are consumed exclusively as executables are built with the oldest possible to ensure broad compatibility. This has been the approach followed for Conan 1.x packages in Conan Center, and not just with regards to libstdc++ but also glibc.

I believe the specific issue here is that the Conan 2.0 packages published in Conan Center are built with gcc11 as the baseline, and thus it does not satisfy the assumption and might be an issue for users who are trying to run those binaries on older versions and consider possible solutions.

Could you provide details as to which os (distro version etc) and compiler versions you are trying to target, and which Docker images you are using for this? Thanks!

[samuel-emrys]

Thanks for the response @jcar87. I've been trying to build my application with gcc10, gcc11, gcc12. I use the conanio/gcc{10,11,12}-ubuntu16.04 docker images to do this. doxygen is built using the conanio/gcc11-ubuntu16.04 image (or something similar), and so all of my builds are failing on conanio/gcc10-ubuntu16.04

I think the fact that conan 2 is only building with a very limited subset of compilers (i.e., only gcc11) is highlighting the deficiency with solution (1) as I outlined as a long term solution. It might be broadly sufficient for what conan center provides in terms of binaries, but in terms of recipes it opens the door to easily creating incompatible packages that won't identify the incompatibility and trigger a rebuild to ensure that a compatible binary is produced.

I've demonstrated a working implementation of what a change in policy might look like in #17051. Unfortunately it seems to be failing the CI builds at the moment because it relies on reading settings.yml to identify the set of compiler versions to compare against, but it can't find the file on the CI machine. Happy to receive guidance on a better approach for this but that discussion is perhaps better placed in that pull request.