search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
975 stars 1.79k forks source link

spdlog spdlog/1.11.0: implicitly changed required fmt without change of own version - broken builds #18134

Open yshurik opened 1 year ago

yshurik commented 1 year ago

Description

spdlog 1.11.0 quietly applied change to change the requirement require from fmt/9.1.0 to fmt/10.0.0 without the upgrade of the version (like 1.11.1)

Now old build recipes become invalid - forced fmt update while a lot was depending on 9.1.0, a lot of object files got linking errors. Old artifacts become invalid.

Can the change be reverted and spd/1.11.1 be introduced with such changed requirement?

conan.io still have wrong dependency referent to 9.1.0 - https://conan.io/center/spdlog?version=1.11.0&revision=d0fdbaa523550b89156084bf42b41c90&tab=dependencies&os

Package and Environment Details

Conan profile

[settings] os=Macos os_build=Macos arch=armv8 arch_build=armv8 compiler=apple-clang compiler.version=14 compiler.libcxx=libc++ build_type=Release [options] [conf] [build_requires] [env]

Steps to reproduce

Try to rebuild existing package which was already successfully built with spdlog/1.11.0

ERROR: Conflict in spdlog/1.11.0: 'spdlog/1.11.0' requires 'fmt/10.0.0' while 'Application/1.0.0@Common+Application/stable' requires 'fmt/9.1.0'. To fix this conflict you need to override the package 'fmt' in your root package.

Logs

Click to expand log ``` ERROR: Conflict in spdlog/1.11.0: 'spdlog/1.11.0' requires 'fmt/10.0.0' while 'Application/1.0.0@Common+Application/stable' requires 'fmt/9.1.0'. To fix this conflict you need to override the package 'fmt' in your root package. ```
Seadex-GmbH commented 1 year ago

This is true and stops the development of other libraries/apps that are using libraries that have spdlog dependencies. I want to build a library that uses seadex-essentials library. And seadex-essentials has a spdlog and fmt dependency.

Now I get the error and it is a show-stopper:

ERROR: Version conflict: spdlog/1.11.0->fmt/10.0.0, seadex-essentials/2.1.3->fmt/9.1.0.

This bug shall have a high priority.

jwidauer commented 1 year ago

This is especially bad, since the fmt version seems to just have been bumped for no apparent reason in this PR. The official spdlog/1.11.0 release, specifically depends on fmt/9.1.0 (reference). So why is it now depending on fmt/10.0.0 here?

jcar87 commented 1 year ago

Hi @yshurik - thank you for raising this issue.

a lot of object files got linking errors. Old artifacts become invalid.

Would you be able to provide us with build logs, and, better yet, a set of reproducible commands where this is happening, so that we can troubleshoot this? The change is not expected to cause those issues, and if there is a version conflict Conan would error out a lot earlier than a build proceeding to linking stage.

Hi @Seadex-GmbH - thank you for reporting this. We are trying to improve our process such that when dependencies are bumped like this, downstream consumers (such as seadex-essentials) are not affected by conflicts. We will proceed to fix this in seadex-essentials in https://github.com/conan-io/conan-center-index/pull/18318 (cc @RubenRBS )

@jwidauer and @yshurik - version bumps of dependencies are allowed in Conan Center and they do not regularly require changing the version of the recipe that has the dependency, thus, it's perfectly commonplace in Conan Center for different revisions of the same recipe (such as spdlog/1.11.0) to have different versions of dependencies. As far as I can see, both vcpkg and homebrew have proceeded in exactly the same way, and their spdlog 1.11.0 is built with fmt 10.0.0 the same as Conan Center. This is done to enable users to use the latest available version of fmt, even if they are also using spdlog, to avoid being constrained a library requiring a specific older version, provided that the library can be rebuilt against a newer version (as is the case here, with the patch applied).

If you would like to protect yourselves from changes in the recipes that affect the dependencies, keep in mind that recipe revisions do change for these sort of changes, and you can always use Conan lockfiles (docs here) to pin this down to a recipe revision where spdlog was using an earlier version of fmt - this is is strongly advised in production scenarios.

The conflict errors by Conan are correct - you need to help Conan choose which version to use to resolve the conflict - which can be done with an override as the error suggests. Conan can then take care of ensuring the packages in your dependency graph are all built with the specific version of fmt. This is unlike the approaches in other package managers or repositories where you are given no choice, and you are pulled to the latest. In Conan Center, we are progressively moving towards using version ranges, and fmt is a good choice for this, as it is known to be backwards compatible (save some minor changes from time to time).

@yshurik - any additional information you can provide regardling linker errors due to mismatching symbols we can look into.

@Seadex-GmbH we will ensure the conflict is resolved in the seadex-essentials recipe

@jwidauer - please let me know if you need more information or guidance regarding different strategies to ensure a safe landing of upstream changes in recipe revisions in Conan Center.

aunsbjerg commented 1 year ago

+1 on this. Took me a while to figure the reason for my CI failing was because fmt was silently bumped to 10.0.0 causing compilation errors in my application. Had to manually override the fmt version in my conanfile to fix it.

I see this in my log:

WARN: spdlog/1.11.0: requirement fmt/10.0.0 overridden by your conanfile to fmt/9.1.0

Event though all spdlog/1.11.0 point to fmt/9.1.0 as far as I can tell.

jcar87 commented 1 year ago

Hi @aunsbjerg - do you have any logs or details on how to reproduce these compilation errors? It would be greatly helpful for us to troubleshoot the issue.

The conflicts and the overrides work in such a way that Conan should ensure that everything is built with compatible versions, precisely to avoid issues at the compilation or linking stage.

aunsbjerg commented 1 year ago

@jcar87 a minimal reproducible setup which compiles fine, but uses the wrong fmt version:

Conanfile.txt:

[requires]
spdlog/1.11.0

[generators]
CMakeDeps
CMakeToolchain

CMakeLists.txt:

cmake_minimum_required(VERSION 3.12)

find_package(spdlog REQUIRED)

add_executable(test
  main.cpp
)

target_link_libraries(test
  PRIVATE
    spdlog::spdlog
)

main.cpp:

#include <spdlog/spdlog.h>

int main()
{
    spdlog::info("Hello, {}!", "World");
    return 0;
}

And build output:

> ~/projects/sandbox/cpp-stuff/build
conan install \                                                                                                                                                                                                   
    --build=missing \
    --install-folder cmake-build-release \
    --settings build_type=Release \
    --profile:build=default \
    ..

Configuration (profile_host):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++11
compiler.version=10
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

conanfile.txt: Installing package
Requirements
    fmt/10.0.0 from 'conancenter' - Cache
    spdlog/1.11.0 from 'conancenter' - Cache
Packages
    fmt/10.0.0:bb317a1f4f7069078e967719a5c585f83cd078c2 - Cache
    spdlog/1.11.0:5db425e61ce3f81a616604dca1d6ae4d11e6513d - Cache

Installing (downloading, building) binaries...
fmt/10.0.0: Already installed!
spdlog/1.11.0: Already installed!
conanfile.txt: Generator 'CMakeToolchain' calling 'generate()'
conanfile.txt: Preset 'release' added to CMakePresets.json. Invoke it manually using 'cmake --preset release'
conanfile.txt: If your CMake version is not compatible with CMakePresets (<3.19) call cmake like: 'cmake <path> -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE=/home/mlaj/projects/sandbox/cpp-stuff/build/cmake-build-release/conan_toolchain.cmake -DCMAKE_POLICY_DEFAULT_CMP0091=NEW -DCMAKE_BUILD_TYPE=Release'
conanfile.txt: Generator txt created conanbuildinfo.txt
conanfile.txt: Generator 'CMakeDeps' calling 'generate()'
conanfile.txt: Aggregating env generators
conanfile.txt: Generated conaninfo.txt
conanfile.txt: Generated graphinfo

> ~/projects/sandbox/cpp-stuff/build
cmake \                                                                                                                                                                                                           
    -GNinja \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_TOOLCHAIN_FILE=cmake-build-release/conan_toolchain.cmake \
    ..

-- Using Conan toolchain: /home/mlaj/projects/sandbox/cpp-stuff/build/cmake-build-release/conan_toolchain.cmake
-- The CXX compiler identification is GNU 10.3.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: Component target declared 'spdlog::spdlog'
-- Conan: Component target declared 'fmt::fmt'
-- Configuring done (0.2s)
-- Generating done (0.0s)
-- Build files have been written to: /home/mlaj/projects/sandbox/cpp-stuff/build

> ~/projects/sandbox/cpp-stuff/build                                                                                     
ninja  

[2/2] Linking CXX executable src/test
jcar87 commented 1 year ago

Hi @aunsbjerg, thank you for your prompt reply. I can see that this is using the same version of fmt that the spdlog recipe currently requires, so it would appear correct, especially if it builds and runs fine. What were you expecting instead, or what issues were you experiencing in CI?

aunsbjerg commented 1 year ago

Hi @aunsbjerg, thank you for your prompt reply. I can see that this is using the same version of fmt that the spdlog recipe currently requires, so it would appear correct, especially if it builds and runs fine. What were you expecting instead, or what issues were you experiencing in CI?

Well, I would expect fmt/9.1.0 as per conan.io but I am honestly confused about what to expect at this point. Doing conan info gives me this:

spdlog/1.11.0
    ID: 5db425e61ce3f81a616604dca1d6ae4d11e6513d
    BuildID: None
    Context: host
    Remote: conancenter=https://center.conan.io
    URL: https://github.com/conan-io/conan-center-index
    Homepage: https://github.com/gabime/spdlog
    License: MIT
    Description: Fast C++ logging library
    Topics: logger, logging, log-filtering, file sink, header-only
    Provides: spdlog
    Recipe: Cache
    Binary: Cache
    Binary remote: conancenter
    Creation date: 2023-06-20 04:52:24 UTC
    Required by:
        conanfile.txt
    Requires:
        fmt/10.0.0

The issue I saw in CI was the my build was suddenly breaking because of a compiler error caused by the bump in fmt version. No changes in my conanfile had been made between runs.

pwqbot commented 1 year ago

@jcar87 can we use version range for fmt?

Shogutora commented 1 year ago

This is still not fixed, causing weird behaviour where in the pipeline build the dependency is 9.1.0 and in local machine it complains about the conflict of spdlog requiring 10.0.0

The recipe revision it was trying to use locally was: 8b4bd1da26e0ec97698369d6224d93bf which does not exist in conan center.

I'm using Artifactory virtual repository as my default remote, this might have something to do with the matter. Upon further inspection I found out that it keeps downloading the aforementioned version of the recipe no matter what.

I just wonder where does this recipe version come from as it is not present in conan center.

EDIT: Per conan-center-index the fmt 10.0.0 dependency is correct, and conan.io has outdated information, meaning that we all must bump fmt-> 10.0.0

jcar87 commented 1 year ago

Hi @Shogutora - I can try to look this up for you. Which reference has a recipe revision of 8b4bd1da26e0ec97698369d6224d93bf? spdlog/1.11.0 or one of the fmt ones?

Edit: from what I can see, 8b4bd1da26e0ec97698369d6224d93bf is the most recent recipe revision available in Conan Center for spdlog/1.11.0:

conan list "spdlog/1.11.0#*" -r conancenter
conancenter
  spdlog
    spdlog/1.11.0
      revisions
        1d2d6e5e3b3c47131995291f7c52fb82 (2022-11-03 13:05:47 UTC)
        51a8dbbfe4ea24e30e57920ce5283690 (2022-11-16 12:12:32 UTC)
        1e0f4eb6338d05e4bd6fcc6bf4734172 (2022-12-08 20:03:44 UTC)
        faa6eb03bd1009bf2070b0c77e4f56a6 (2023-02-09 11:02:15 UTC)
        1f94812dde789fba690ebc10e9613e61 (2023-03-27 21:00:48 UTC)
        d0fdbaa523550b89156084bf42b41c90 (2023-04-12 21:47:59 UTC)
        8b4bd1da26e0ec97698369d6224d93bf (2023-06-20 02:52:29 UTC)
Shogutora commented 1 year ago

Hi @jcar87 yes, so the disparency is with the recipes available in https://conan.io/center/spdlog and conan-center index. So we should just trust what the actual recipes give us -> fmt 10.0.0.

yshurik commented 1 year ago

Hi @yshurik - thank you for raising this issue.

a lot of object files got linking errors. Old artifacts become invalid.

Would you be able to provide us with build logs, and, better yet, a set of reproducible commands where this is happening, so that we can troubleshoot this? The change is not expected to cause those issues, and if there is a version conflict Conan would error out a lot earlier than a build proceeding to linking stage.

Hi @Seadex-GmbH - thank you for reporting this. We are trying to improve our process such that when dependencies are bumped like this, downstream consumers (such as seadex-essentials) are not affected by conflicts. We will proceed to fix this in seadex-essentials in #18318 (cc @RubenRBS )

@jwidauer and @yshurik - version bumps of dependencies are allowed in Conan Center and they do not regularly require changing the version of the recipe that has the dependency, thus, it's perfectly commonplace in Conan Center for different revisions of the same recipe (such as spdlog/1.11.0) to have different versions of dependencies. As far as I can see, both vcpkg and homebrew have proceeded in exactly the same way, and their spdlog 1.11.0 is built with fmt 10.0.0 the same as Conan Center. This is done to enable users to use the latest available version of fmt, even if they are also using spdlog, to avoid being constrained a library requiring a specific older version, provided that the library can be rebuilt against a newer version (as is the case here, with the patch applied).

If you would like to protect yourselves from changes in the recipes that affect the dependencies, keep in mind that recipe revisions do change for these sort of changes, and you can always use Conan lockfiles (docs here) to pin this down to a recipe revision where spdlog was using an earlier version of fmt - this is is strongly advised in production scenarios.

The conflict errors by Conan are correct - you need to help Conan choose which version to use to resolve the conflict - which can be done with an override as the error suggests. Conan can then take care of ensuring the packages in your dependency graph are all built with the specific version of fmt. This is unlike the approaches in other package managers or repositories where you are given no choice, and you are pulled to the latest. In Conan Center, we are progressively moving towards using version ranges, and fmt is a good choice for this, as it is known to be backwards compatible (save some minor changes from time to time).

@yshurik - any additional information you can provide regardling linker errors due to mismatching symbols we can look into.

@Seadex-GmbH we will ensure the conflict is resolved in the seadex-essentials recipe

@jwidauer - please let me know if you need more information or guidance regarding different strategies to ensure a safe landing of upstream changes in recipe revisions in Conan Center.

The main issue about this is that we can not reproduce exactly same builds - to get libraries and executables which were successfully produced just weeks ago.

Because before it was strict dependency on fmt/9.1.0 then all other libraries and modules were compiled in also have strict dependency to fmt/9.1.0. Now due to this "hidden" update all other libraries and modules which were built with fmt/9.1.0 can not be linked vs our applications because of linking errors (of course old modules are linked vs fmt/9.1.0)

It is an obstacle to rebuild all chain and re-release and re-test all modules which were explicitly and implicitly dependent to fmt/9.1.0 as it is huge project with a lot of dependencies. So this is really CI/CD breakage with no way to revert to previously working state.

Yes conan has /stable /beta channels, and what you just released and mention is supposed be policy for working in /beta channel.

But this is conan.io package repository and all users of the repository consider it as /stable channel with a way to reproduce the build if it has referred specific version. So you should not do such update without a bump of the version change to make the work with the versions of this conan module explicit.

jcar87 commented 1 year ago

Now due to this "hidden" update all other libraries and modules which were built with fmt/9.1.0 can not be linked vs our applications because of linking errors (of course old modules are linked vs fmt/9.1.0)

These are the logs I'm asking about. We really can't investigate any further without actual logs, and we can only speculate what the issue is.

It is an obstacle to rebuild all chain and re-release and re-test all modules which were explicitly and implicitly dependent to fmt/9.1.0.

We really need to see actual build logs with the errors to ascertain what is happening, so that we can provide guidance and if there is something wrong on our part, avoid it in the future. We have already explained above that you can use an override to make sure fmt/9.1.0 is used across the entire dependency graph - which judging by the details you are providing ought to be enough to avoid re-building anything.

I can see two possible issues here:

So this is really CI/CD breakage with no way to revert to previously working state. The recipe revision where spdlog/1.11.0 depends on fmt/9.1.0 is still available on the Conan Center remote, and you are free to use it. When you have a working set of dependencies, it is highly advised to generate and save a lockfile: https://docs.conan.io/2.0/tutorial/versioning/lockfiles.html.

yshurik commented 1 year ago

Now due to this "hidden" update all other libraries and modules which were built with fmt/9.1.0 can not be linked vs our applications because of linking errors (of course old modules are linked vs fmt/9.1.0)

These are the logs I'm asking about. We really can't investigate any further without actual logs, and we can only speculate what the issue is.

It is an obstacle to rebuild all chain and re-release and re-test all modules which were explicitly and implicitly dependent to fmt/9.1.0.

We really need to see actual build logs with the errors to ascertain what is happening, so that we can provide guidance and if there is something wrong on our part, avoid it in the future. We have already explained above that you can use an override to make sure fmt/9.1.0 is used across the entire dependency graph - which judging by the details you are providing ought to be enough to avoid re-building anything.

I can see two possible issues here:

  • Different parts of the graph depend on different versions of fmt, which causes Conan to produce a conflict when the recipe revision of spdlog/1.11.0 is updated to depend on the newer fmt. The solution to this is an override from the top-level, and you can override fmt to exactly the older version so that all the binaries you had before are still used and link correctly. Here there aren't linker or compiler errors, in the sense that conan install will fail to resolve the graph. If the graph is then resolved via overrides
  • Parts of the graph depend on fmt/9.1.0, but they do not express it in the requirements, and instead use it while relying on it coming transitively from a different dependency, such as spdlog. This is leaving the fmt version to be resolved somewhere else - and if it is ever updated to a version that is source-incompatible or link-incompatible, then you would get errors. We cannot stress enough: any component that depends on fmt should express the requirement on fmt, and if it needs a specific version, it should list that.

So this is really CI/CD breakage with no way to revert to previously working state. The recipe revision where spdlog/1.11.0 depends on fmt/9.1.0 is still available on the Conan Center remote, and you are free to use it. When you have a working set of dependencies, it is highly advised to generate and save a lockfile: https://docs.conan.io/2.0/tutorial/versioning/lockfiles.html.

The typical linking errors were like this (just showing first 3, there is a long list): /usr/bin/ld: /root/.conan/data/OpenId/1.4.7/Common+OpenId/stable/package/21c61c22f5eeef3ed54bb7a03a1648265f5af066/lib/libOpenId.a(Common.o): in functionstd::cxx11::basic_string<char, std::char_traits, std::allocator > fmt::v9::format<boost::core::basic_string_view, std::cxx11::basic_string<char, std::char_traits, std::allocator > >(fmt::v9::basic_format_string<char, fmt::v9::type_identity<boost::core::basic_string_view >::type, fmt::v9::type_identity<std::cxx11::basic_string<char, std::char_traits, std::allocator > >::type>, boost::core::basic_string_view&&, std::cxx11::basic_string<char, std::chartraits, std::allocator >&&)': /root/.conan/data/fmt/9.1.0//_/package/8fbeded8e81bfcbd1fcdca301b03a876d262e652/include/fmt/core.h:3206: undefined reference to fmt::v9::vformat[abi:cxx11](fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >)' /usr/bin/ld: /root/.conan/data/OpenId/1.4.7/Common+OpenId/stable/package/21c61c22f5eeef3ed54bb7a03a1648265f5af066/lib/libOpenId.a(OpenId.o): in functionstd::__cxx11::basic_string<char, std::char_traits, std::allocator > fmt::v9::format<std::basic_string_view<char, std::char_traits > const&>(fmt::v9::basic_format_string<char, fmt::v9::type_identity<std::basic_string_view<char, std::char_traits > const&>::type>, std::basic_string_view<char, std::chartraits > const&)': /root/.conan/data/fmt/9.1.0//_/package/8fbeded8e81bfcbd1fcdca301b03a876d262e652/include/fmt/core.h:3206: undefined reference to fmt::v9::vformat[abi:cxx11](fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >)' /usr/bin/ld: /root/.conan/data/OpenId/1.4.7/Common+OpenId/stable/package/21c61c22f5eeef3ed54bb7a03a1648265f5af066/lib/libOpenId.a(OpenId.o): in functionfmt::v9::detail::error_handler::on_error(char const*)': `

jcar87 commented 1 year ago

Hi @yshurik , thank you for providing logs. We would need the following information:

Thanks!

goncalves-alex commented 1 year ago

As a workaround you can override the fmt version in your project with this:

self.requires("fmt/9.1.0", override=True)

GooRoo commented 2 months ago

So, I haven't touch a project with explicit dependencies to fmt/9.1.0 and spdlog/1.11.0 for about a year. And now I tried to compile it, but I can't anymore.

This is absolutely cursed, I must say.