search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.25k stars 980 forks source link

[question] Filtering options for package to get path to local cache #16544

Closed mzbro closed 4 months ago

mzbro commented 4 months ago

What is your question?

Hi! In my CI i need to add boost libs to LD_LIBRARY_PATH, since those are missing, which causes this error during build:

/home/jenkins/workspace/super-project/super-googletests: error while loading shared libraries: libboost_filesystem.so.1.82.0: cannot open shared object file: No such file or directory
CMake Error at /opt/cmake-3.27.7-linux-x86_64/share/cmake-3.27/Modules/GoogleTestAddTests.cmake:112 (message):
  Error running test executable.

    Path: '/home/jenkins/workspace/super-project/super-googletests'
    Result: 127
    Output:      

Call Stack (most recent call first):
  /opt/cmake-3.27.7-linux-x86_64/share/cmake-3.27/Modules/GoogleTestAddTests.cmake:225 (gtest_discover_tests_impl)

I have read issue 14974 - this misses information on filtering options.

In conanfile.py i require boost by method:

    default_options = {"shared": False, "fPIC": True, "boost/*:shared": True}

    def requirements(self):
        self.requires("boost/1.82.0", transitive_libs=True, transitive_headers=True)

I have also tried adding run trait, which does not resolve this issue:

    default_options = {"shared": False, "fPIC": True, "boost/*:shared": True}

    def requirements(self):
        self.requires("boost/1.82.0", transitive_libs=True, transitive_headers=True, run=True)

Edit: added default_options for requirements information.

To patch LD_LIBRARY_PATH with boost shared libs I use:

pkg_id=$(conan list "boost/1.82.0#*:*#latest" -fo '&:shared=True' -fp ${BUILD_PROFILE} -f compact | grep 'boost/1.82.0#' | sed -n '2{p;q}' | awk {'print $1'})
export LD_LIBRARY_PATH="$(conan cache path "${pkg_id}")"/lib

Unfortunatelly, filtering by option shared does not work - sometimes above results in static libs in my LD_LIBRARY_PATH variable.

To make sure, I cleaned whole cache with:

conan remove "*" -c
conan cache clean -s -b -t -d "*"

Next, I installed all 4 packages from our conan-local remote (Release/Debug - both with static and shared options), which is in self-hosted Artifactory:

conan install --requires="boost/1.82.0" -r=conan-local -o '&:shared=False' -pr=default
conan install --requires="boost/1.82.0" -r=conan-local -o '&:shared=True' -pr=default
conan install --requires="boost/1.82.0" -r=conan-local -o '&:shared=False' -pr=Debug
conan install --requires="boost/1.82.0" -r=conan-local -o '&:shared=True' -pr=Debug

Above packages where build from conancenter with --build=missing option, and uploaded to conan-local remote.

Now I have 4 revisions in cache:

» conan list "boost/1.82.0#*:*" -c -f compact                                                                                                                                                                                mzbroszczyk@K-15636-bookworm
Local Cache
  boost/1.82.0
    boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801%1717769236.89 (2024-06-07 14:07:16 UTC)
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:11140526e509fdbb50ad0b0a14039671adc0a7bd
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:35b9aa0f4ef9acaa7fa0359a2c86b340a0c26906
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:029610a532978ba21a4d9193e4b39b4e66844cf0, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:1a9fda660de0c747d126d629ab86cd194228796f, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:1a9fda660de0c747d126d629ab86cd194228796f
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:9a629a48ecde3fd665c78689021583735fe6efab
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:4530633a3406a354d9c70a71124250144067de22, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:eb634d99a593f7975428678fe6d715b0e3566434
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z

And now, this is what confuses me: when I filter by shared option, it doesn't work:

» conan list "boost/1.82.0#*:*" -c -f compact -fo '&:shared=True'                                                                                                                                                            mzbroszczyk@K-15636-bookworm
Local Cache
  boost/1.82.0
    boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801%1717769236.89 (2024-06-07 14:07:16 UTC)
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:11140526e509fdbb50ad0b0a14039671adc0a7bd
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:35b9aa0f4ef9acaa7fa0359a2c86b340a0c26906
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:029610a532978ba21a4d9193e4b39b4e66844cf0, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:1a9fda660de0c747d126d629ab86cd194228796f, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:1a9fda660de0c747d126d629ab86cd194228796f
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:9a629a48ecde3fd665c78689021583735fe6efab
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:4530633a3406a354d9c70a71124250144067de22, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:eb634d99a593f7975428678fe6d715b0e3566434
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z

However, filtering by profile works fine:

» conan list "boost/1.82.0#*:*" -c -f compact -fo '&:shared=True' -fp=Debug                                                                                                                                                  mzbroszczyk@K-15636-bookworm
Local Cache
  boost/1.82.0
    boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801%1717769236.89 (2024-06-07 14:07:16 UTC)
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:11140526e509fdbb50ad0b0a14039671adc0a7bd
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:35b9aa0f4ef9acaa7fa0359a2c86b340a0c26906
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:029610a532978ba21a4d9193e4b39b4e66844cf0, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:1a9fda660de0c747d126d629ab86cd194228796f, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:1a9fda660de0c747d126d629ab86cd194228796f

Am I missing any magic here? I've seen docs about deployers, but I'm not sure I want to use it. All I need is path to local package in cache, something which was very easy and straight-forward in conan 1.x.

Have you read the CONTRIBUTING guide?

AbrilRBS commented 4 months ago

Hi @mzbro thanks a lot for your question, I have a minor comment for this:

And now, this is what confuses me: when I filter by shared option, it doesn't work:

This is because you're scoping the option with "&:", changing it to -fo="shared=True" should work. I've filled https://github.com/conan-io/conan/issues/16545 for one of the UX issues that I've noticed might have confussed you :)

mzbro commented 4 months ago

@RubenRBS Thank you for your answer. Unfortunately this also doesn't work:

» conan list "boost/1.82.0#*:*" -c -f compact -fo="shared=True"                                                                                                     mzbroszczyk@K-15636-bookworm
WARN: legacy: Unscoped option definition is ambiguous.
Use '&:shared=True' to refer to the current package.
Use '*:shared=True' or other pattern if the intent was to apply to dependencies
Local Cache
  boost/1.82.0
    boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801%1717769236.89 (2024-06-07 14:07:16 UTC)
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:11140526e509fdbb50ad0b0a14039671adc0a7bd
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:35b9aa0f4ef9acaa7fa0359a2c86b340a0c26906
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:029610a532978ba21a4d9193e4b39b4e66844cf0, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:1a9fda660de0c747d126d629ab86cd194228796f, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:1a9fda660de0c747d126d629ab86cd194228796f
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:9a629a48ecde3fd665c78689021583735fe6efab
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): shared=True
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:4530633a3406a354d9c70a71124250144067de22, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:5bc851010eb7b707e5cb2e24cb8ccf0f27989fa9
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:eb634d99a593f7975428678fe6d715b0e3566434
        settings: Linux, x86_64, Release, gcc, libstdc++11, 12
        options(diff): fPIC=True, shared=False
        requires: bzip2/1.0.Z, libbacktrace/cci, zlib/1.3.Z

I use latest conan version 2.4.1

AbrilRBS commented 4 months ago

Sorry it seems like I missremembered, you're actually supposed to scope it! But you should specify *: as the scope - as this was unclear even to me, I'll have a chat with the team as to how we can better communicate it :)

mzbro commented 4 months ago

Ok, scope *: solved my problem, thanks a lot :-)

» conan list "boost/1.82.0#latest:*" -c -f compact -fo '*:shared=True' -fp=Debug
Local Cache
  boost/1.82.0
    boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801%1717769236.89 (2024-06-07 14:07:16 UTC)
      boost/1.82.0#47e7b0c3ddccedf39a3baf57a7e61801:35b9aa0f4ef9acaa7fa0359a2c86b340a0c26906
        settings: Linux, x86_64, Debug, gcc, libstdc++11, 12
        options(diff): 
        requires: bzip2/1.0.8#457c272f7da34cb9c67456dd217d36c4:029610a532978ba21a4d9193e4b39b4e66844cf0, libbacktrace/cci.20210118#ec1aa63bbc10145c6a299e68e711670c:1a9fda660de0c747d126d629ab86cd194228796f, zlib/1.3.1#f52e03ae3d251dec704634230cd806a2:1a9fda660de0c747d126d629ab86cd194228796f