search

conan-io / conan

Conan - The open-source C and C++ package manager
https://conan.io
MIT License
8.21k stars 979 forks source link

[question] How to disable search of dependency's test binary packages #13439

Open lo1ol opened 1 year ago

lo1ol commented 1 year ago

What is your question?

I find out that conan search for test binary package of dependencies. It's may lead to long search of unnecessary packages, which will be skipped after search.

Furthermore, I have a problem if packages depends on each other. I have rtengine package, which uses pkcs11ecp package for tests. After some time I add rtengine to pkcs11ecp for test. And after that update rtengine to new version of pkcs11ecp. Install works fine, but export-pkg looks for unnecessary packages and at the final can't find necessary binary for pkcs11ecp.

rtengine conanfile:

from conan import ConanFile
from conan.tools.files import copy

from os import path

class RtengineConanfile(ConanFile):
    settings = "os", "compiler", "arch", "build_type", "libc"
    options = {"shared": [True, False], "restricted": [True, False], "country": ["ru", "kaz"]}
    default_options = {"shared": True, "restricted": False, "country": "ru"}

    python_requires = "rutokenconanfile/0.29@rutoken/testing"
    python_requires_extend = "rutokenconanfile.RutokenConanFile"

    def build_requirements(self):
        self.test_requires("boost-mt/1.76.0-e7488ca@rutoken/experimental")
        self.test_requires("pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing", options={"country": self.options.country})
        self.test_requires("openssl/3.0.7-cb192d2@rutoken/experimental")

    def requirements(self):
        self.requires("rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental", visible=not self.options.shared)

    def configure(self):
        self.settings.rm_safe("compiler.cppstd")

        if self.settings.os == "Windows":
            self.settings.compiler.runtime = "static"
            self.settings.compiler.runtime_type = self.settings.build_type

    def package(self):
        copy(self, "*", path.join(self.build_folder, "shared" if self.options.shared else "static"), self.package_folder)

updated pkcs11ecp conanfile (old version depends on nothing)

from conan import ConanFile
from conan.tools.files import copy

from os import path

class RtPkcs11EcpConanfile(ConanFile):
    settings = "os", "compiler", "arch", "build_type", "libc"
    options = {"shared": [True, False], "sysformat": [True, False], "crypto": [True, False], "country": ["ru", "kaz"], "use_cprocsp": [True, False]}
    default_options = {"shared": True, "sysformat": False, "crypto": False, "country": "ru"}

    python_requires = "rutokenconanfile/0.29@rutoken/testing"
    python_requires_extend = "rutokenconanfile.RutokenConanFile"

    def configure(self):
        self.settings.rm_safe("compiler.cppstd")

        if self.settings.os == "Windows":
            self.settings.compiler.runtime = "static"
            self.settings.compiler.runtime_type = self.settings.build_type

        self.options.use_cprocsp = self.settings.os in ["Windows", "Linux", "FreeBSD", "Macos"] and self.settings.arch in  ["x86", "x86_64"]

    def build_requirements(self):
        self.test_requires("tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing", options={"country": self.options.country})

        if self.options.use_cprocsp:
            self.test_requires("cprocsp/5.0.11732@rutoken/experimental")

        if self.settings.os in ["FreeBSD", "Linux"]:
            self.test_requires("pcsclite/1.9.5@rutoken/testing#a50e78deffa149d380cc0a88381cf124")
        elif self.settings.os == "Android":
            self.test_requires("pcsc-android/v1.10.0-12-gf5d49a60@rutoken/testing")
            self.test_requires("rtservice/v1.10.0-12-gf5d49a60@rutoken/testing")
        elif self.settings.os == "iOS":
            self.test_requires("pcsc-ios/v4.0.0@rutoken/experimental")
        elif self.settings.os == "Macos":
            self.test_requires("osx-ccid/e560ac36b10e96df43af4ee22d0aa6299f89b327@rutoken/testing", options={"country": self.options.country})

    def requirements(self):
        visible = not self.options.shared
        self.requires("openssl/3.0.7-cb192d2@rutoken/experimental", visible=visible, options={"shared": False})
        self.requires("rtengine/v3.1.1-5-g5535964@rutoken/experimental", visible=visible,
                      options={"country": self.options.country, "restricted": self.options.country == "ru", "shared": False})

    def package(self):
        if self.options.shared:
            copy(self, "*", path.join(self.build_folder, "shared"), self.package_folder)
        else:
            copy(self, "*", path.join(self.build_folder, "static"), self.package_folder)

    def package_id(self):
        del self.info.options.use_cprocsp

conan install works fine, but search for unnecessary packages (conan search for test_dependencies for test_dependencies) and has bad output(boost-mt is skipped but actually downloaded):

 conan install --name kek --version 0.0.1 -pr glibc-x86_64-gcc11 -of kek -s build_type=Debug .

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
libc=glibc
libc.version=2.12
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

rutokenconanfile/0.29@rutoken/testing: Not found in local cache, looking in remotes...
rutokenconanfile/0.29@rutoken/testing: Checking remote: conancenter
rutokenconanfile/0.29@rutoken/testing: Checking remote: rutoken
rutokenconanfile/0.29@rutoken/testing: Downloaded recipe revision 25973501fa393aae9945419209b747fe

======== Computing dependency graph ========
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Not found in local cache, looking in remotes...
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Checking remote: conancenter
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Checking remote: rutoken
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Downloaded recipe revision 1f4153c5c04cedc619e3faa2c9e025e8
rutokenconanfile/0.26@rutoken/testing: Not found in local cache, looking in remotes...
rutokenconanfile/0.26@rutoken/testing: Checking remote: conancenter
rutokenconanfile/0.26@rutoken/testing: Checking remote: rutoken
rutokenconanfile/0.26@rutoken/testing: Downloaded recipe revision b1257b9a92f36b4c09fa008f25a292ca
cprocsp/5.0.11732@rutoken/experimental: Not found in local cache, looking in remotes...
cprocsp/5.0.11732@rutoken/experimental: Checking remote: conancenter
cprocsp/5.0.11732@rutoken/experimental: Checking remote: rutoken
cprocsp/5.0.11732@rutoken/experimental: Downloaded recipe revision 75066d2cfa02a28f12b562ed4dae876d
rutokenconanfile/0.25@rutoken/testing: Not found in local cache, looking in remotes...
rutokenconanfile/0.25@rutoken/testing: Checking remote: conancenter
rutokenconanfile/0.25@rutoken/testing: Checking remote: rutoken
rutokenconanfile/0.25@rutoken/testing: Downloaded recipe revision 2933131cd0b258b5e47c6fbee93c7bb0
boost-mt/1.76.0-e7488ca@rutoken/experimental: Not found in local cache, looking in remotes...
boost-mt/1.76.0-e7488ca@rutoken/experimental: Checking remote: conancenter
boost-mt/1.76.0-e7488ca@rutoken/experimental: Checking remote: rutoken
boost-mt/1.76.0-e7488ca@rutoken/experimental: Downloaded recipe revision 766f131873c73bc3cde079bff2bf1754
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Not found in local cache, looking in remotes...
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Checking remote: conancenter
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Checking remote: rutoken
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Downloaded recipe revision c0a84b2701d504989b1bbf53a4663f79
openssl/3.0.7-cb192d2@rutoken/experimental: Not found in local cache, looking in remotes...
openssl/3.0.7-cb192d2@rutoken/experimental: Checking remote: conancenter
openssl/3.0.7-cb192d2@rutoken/experimental: Checking remote: rutoken
openssl/3.0.7-cb192d2@rutoken/experimental: Downloaded recipe revision c691c52600664b10b7477f3ca576ee9c
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Not found in local cache, looking in remotes...
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Checking remote: conancenter
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Checking remote: rutoken
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Downloaded recipe revision 6ab88a1007911bbc294a0ca014658a40
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Not found in local cache, looking in remotes...
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Checking remote: conancenter
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Checking remote: rutoken
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Downloaded recipe revision 68d7e9dfbe6a53922a47ec4a242016c2
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: Not found in local cache, looking in remotes...
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: Checking remote: conancenter
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: Checking remote: rutoken
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: Downloaded recipe revision c1280aa905c0fe5ef843c05535fb05a1
pcsclite/1.9.5@rutoken/testing: Not found in local cache, looking in remotes...
pcsclite/1.9.5@rutoken/testing: Checking remote: conancenter
pcsclite/1.9.5@rutoken/testing: Checking remote: rutoken
pcsclite/1.9.5@rutoken/testing: Downloaded recipe revision a50e78deffa149d380cc0a88381cf124
Graph root
    conanfile.py (kek/0.0.1): /home/lo1ol/Documents/work/rtengine/conanfile.py
Requirements
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8 - Downloaded (rutoken)
Test requirements
    boost-mt/1.76.0-e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754 - Cache
    cprocsp/5.0.11732@rutoken/experimental#75066d2cfa02a28f12b562ed4dae876d - Cache
    openssl/3.0.7-cb192d2@rutoken/experimental#c691c52600664b10b7477f3ca576ee9c - Cache
    pcsclite/1.9.5@rutoken/testing#a50e78deffa149d380cc0a88381cf124 - Cache
    pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing#68d7e9dfbe6a53922a47ec4a242016c2 - Cache
    pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79 - Downloaded (rutoken)
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8 - Cache
    rtengine/v3.1.1-5-g5535964@rutoken/experimental#6ab88a1007911bbc294a0ca014658a40 - Downloaded (rutoken)
    tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing#c1280aa905c0fe5ef843c05535fb05a1 - Downloaded (rutoken)
Python requires
    rutokenconanfile/0.25@rutoken/testing#2933131cd0b258b5e47c6fbee93c7bb0 - Downloaded (rutoken)
    rutokenconanfile/0.26@rutoken/testing#b1257b9a92f36b4c09fa008f25a292ca - Downloaded (rutoken)
    rutokenconanfile/0.29@rutoken/testing#25973501fa393aae9945419209b747fe - Cache

======== Computing necessary packages ========
boost-mt/1.76.0-e7488ca@rutoken/experimental: Checking 50 compatible configurations:
boost-mt/1.76.0-e7488ca@rutoken/experimental: '4115bd5b2ad61fe73c9251212dce1dca55df9b24': build_type=Release
boost-mt/1.76.0-e7488ca@rutoken/experimental: Main binary package '042da963fb7d3e363c64266822c0ed9890d8c44b' missing. Using compatible package '4115bd5b2ad61fe73c9251212dce1dca55df9b24'
openssl/3.0.7-cb192d2@rutoken/experimental: Checking 13 compatible configurations:
openssl/3.0.7-cb192d2@rutoken/experimental: '9604cbbe3ab4d7a9796f4862b97a22492749a0c2': build_type=Release
openssl/3.0.7-cb192d2@rutoken/experimental: Main binary package 'ce8dbdeab63ab1f0c7ace120ebfd3611aa5dd75f' missing. Using compatible package '9604cbbe3ab4d7a9796f4862b97a22492749a0c2'
openssl/3.0.7-cb192d2@rutoken/experimental: Checking 13 compatible configurations:
openssl/3.0.7-cb192d2@rutoken/experimental: 'd26d4ad84e7d0d9ccde5c37a6514288f50e44efd': build_type=Release
openssl/3.0.7-cb192d2@rutoken/experimental: Main binary package '3a59bc00c13965211a71d35a72250fa70a09ccfb' missing. Using compatible package 'd26d4ad84e7d0d9ccde5c37a6514288f50e44efd'
pcsclite/1.9.5@rutoken/testing: Checking 13 compatible configurations:
pcsclite/1.9.5@rutoken/testing: '98981384b21184d62f24f26a66dd865b2fd3197c': build_type=Release
pcsclite/1.9.5@rutoken/testing: Main binary package '222d94c573a4f4ae5eaee526bc5b3247046770f4' missing. Using compatible package '98981384b21184d62f24f26a66dd865b2fd3197c'
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Checking 13 compatible configurations:
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: '426334e6c2a31bb34a190b48c3114ef5d7cb9140': build_type=Release
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Main binary package '090e2a50985ed8afc712ddce15c945dbe963b810' missing. Using compatible package '426334e6c2a31bb34a190b48c3114ef5d7cb9140'
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Checking 13 compatible configurations:
rtengine/v3.1.1-5-g5535964@rutoken/experimental: 'd1a1e4b7534bbfc447cde6cf7a56191051580ea1': build_type=Release
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Main binary package '65abe93129f3465e8314430308210bd88e0dbd10' missing. Using compatible package 'd1a1e4b7534bbfc447cde6cf7a56191051580ea1'
Requirements
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8:426334e6c2a31bb34a190b48c3114ef5d7cb9140#af57376943cf3eedd16daa20d83faa5e - Download (rutoken)
Test requirements
    boost-mt/1.76.0-e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754:4115bd5b2ad61fe73c9251212dce1dca55df9b24#27200b190241c7a40892d7b861ab44ae - Skip
    cprocsp/5.0.11732@rutoken/experimental#75066d2cfa02a28f12b562ed4dae876d:f2d37cd7199b4fc0a2b397e64f1ae8247612826e#3a1dad8b516b04057e884a80759d132e - Skip
    openssl/3.0.7-cb192d2@rutoken/experimental#c691c52600664b10b7477f3ca576ee9c:9604cbbe3ab4d7a9796f4862b97a22492749a0c2#1d846fcd9aa8288878247144a451edb5 - Skip
    openssl/3.0.7-cb192d2@rutoken/experimental#c691c52600664b10b7477f3ca576ee9c:d26d4ad84e7d0d9ccde5c37a6514288f50e44efd#0b6c0911ab9ee6851314c12bd89168d0 - Download (rutoken)
    pcsclite/1.9.5@rutoken/testing#a50e78deffa149d380cc0a88381cf124:98981384b21184d62f24f26a66dd865b2fd3197c#5e7a4d9580d524973726981151c69944 - Skip
    pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing#68d7e9dfbe6a53922a47ec4a242016c2:784e657aa1b7f1a719ee85805183271a87127175#f8ed13115b6767a8a848c678e1253c75 - Skip
    pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79:435e75f0dbc065a8a7a66009de0c7c625aaf16cc#81d31254a330ea2f51c7f6f091dff66b - Download (rutoken)
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8:426334e6c2a31bb34a190b48c3114ef5d7cb9140#af57376943cf3eedd16daa20d83faa5e - Skip
    rtengine/v3.1.1-5-g5535964@rutoken/experimental#6ab88a1007911bbc294a0ca014658a40:d1a1e4b7534bbfc447cde6cf7a56191051580ea1#e7c3d8500483b424a8ee4ec67f354475 - Skip
    tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing#c1280aa905c0fe5ef843c05535fb05a1:f33bea03e0bf2394aee7ddebe87f0cc1795187e1#013156f93803be3acb55544f840fa2b7 - Skip

======== Installing packages ========

-------- Downloading 4 packages --------
boost-mt/1.76.0-e7488ca@rutoken/experimental: Retrieving package 4115bd5b2ad61fe73c9251212dce1dca55df9b24 from remote 'rutoken' 
Downloading conanmanifest.txt
Downloading conan_package.tgz
Decompressing conan_package.tgz
boost-mt/1.76.0-e7488ca@rutoken/experimental: Package installed 4115bd5b2ad61fe73c9251212dce1dca55df9b24
boost-mt/1.76.0-e7488ca@rutoken/experimental: Downloaded package revision 27200b190241c7a40892d7b861ab44ae
openssl/3.0.7-cb192d2@rutoken/experimental: Retrieving package d26d4ad84e7d0d9ccde5c37a6514288f50e44efd from remote 'rutoken' 
Downloading conan_package.tgz
Decompressing conan_package.tgz
openssl/3.0.7-cb192d2@rutoken/experimental: Package installed d26d4ad84e7d0d9ccde5c37a6514288f50e44efd
openssl/3.0.7-cb192d2@rutoken/experimental: Downloaded package revision 0b6c0911ab9ee6851314c12bd89168d0
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Retrieving package 426334e6c2a31bb34a190b48c3114ef5d7cb9140 from remote 'rutoken' 
Downloading conan_package.tgz
Decompressing conan_package.tgz
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Package installed 426334e6c2a31bb34a190b48c3114ef5d7cb9140
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Downloaded package revision af57376943cf3eedd16daa20d83faa5e
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Retrieving package 435e75f0dbc065a8a7a66009de0c7c625aaf16cc from remote 'rutoken' 
Downloading conan_package.tgz
Decompressing conan_package.tgz
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Package installed 435e75f0dbc065a8a7a66009de0c7c625aaf16cc
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Downloaded package revision 81d31254a330ea2f51c7f6f091dff66b

======== Finalizing install (deploy, generators) ========
conanfile.py (kek/0.0.1): Calling generate()
conanfile.py (kek/0.0.1): Generators folder: /home/lo1ol/Documents/work/rtengine/kek
Install finished succesfully

but export-pkg looks for all compatible packages and can't find pkcs11ecp:

conan export-pkg --name kek --version 0.0.1 -pr glibc-x86_64-gcc11 -of kek -s build_type=Debug .

======== Exporting recipe to the cache ========
kek/0.0.1: Exporting package recipe: /home/lo1ol/Documents/work/rtengine/conanfile.py
kek/0.0.1: Copied 1 '.py' file: conanfile.py
kek/0.0.1: Exported to cache folder: /home/lo1ol/.conan2/p/kek84681448ad62c/e
kek/0.0.1: Exported: kek/0.0.1#e86f8ac25d946b5856e021b4237c46a4 (2023-03-14 07:57:28 UTC)

======== Computing dependency graph ========
Graph root
    conanfile.py (kek/0.0.1): /home/lo1ol/Documents/work/rtengine/conanfile.py
Requirements
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8 - Cache
Test requirements
    boost-mt/1.76.0-e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754 - Cache
    cprocsp/5.0.11732@rutoken/experimental#75066d2cfa02a28f12b562ed4dae876d - Cache
    openssl/3.0.7-cb192d2@rutoken/experimental#c691c52600664b10b7477f3ca576ee9c - Cache
    pcsclite/1.9.5@rutoken/testing#a50e78deffa149d380cc0a88381cf124 - Cache
    pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing#68d7e9dfbe6a53922a47ec4a242016c2 - Cache
    pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79 - Cache
    rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8 - Cache
    rtengine/v3.1.1-5-g5535964@rutoken/experimental#6ab88a1007911bbc294a0ca014658a40 - Cache
    tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing#c1280aa905c0fe5ef843c05535fb05a1 - Cache
Python requires
    rutokenconanfile/0.25@rutoken/testing#2933131cd0b258b5e47c6fbee93c7bb0 - Cache
    rutokenconanfile/0.26@rutoken/testing#b1257b9a92f36b4c09fa008f25a292ca - Cache
    rutokenconanfile/0.29@rutoken/testing#25973501fa393aae9945419209b747fe - Cache

======== Computing necessary packages ========
boost-mt/1.76.0-e7488ca@rutoken/experimental: Checking 50 compatible configurations:
boost-mt/1.76.0-e7488ca@rutoken/experimental: '4115bd5b2ad61fe73c9251212dce1dca55df9b24': build_type=Release
boost-mt/1.76.0-e7488ca@rutoken/experimental: Main binary package '042da963fb7d3e363c64266822c0ed9890d8c44b' missing. Using compatible package '4115bd5b2ad61fe73c9251212dce1dca55df9b24'
openssl/3.0.7-cb192d2@rutoken/experimental: Checking 13 compatible configurations:
openssl/3.0.7-cb192d2@rutoken/experimental: '9604cbbe3ab4d7a9796f4862b97a22492749a0c2': build_type=Release
openssl/3.0.7-cb192d2@rutoken/experimental: 'eecfb0029a97f73fc8850870ea1b888876ac6498': compiler.version=11.3
openssl/3.0.7-cb192d2@rutoken/experimental: 'a0e226842049c89c3d79cb64395076ba92a8bba5': compiler.version=11.2
openssl/3.0.7-cb192d2@rutoken/experimental: '007b1f74046a2218c414c143fe6582309ec534e8': compiler.version=11.1
openssl/3.0.7-cb192d2@rutoken/experimental: 'f367484addf5e44fcc62fb7ec7c78b0f54320446': compiler.version=10
openssl/3.0.7-cb192d2@rutoken/experimental: '3eeddd4635c9f0ea0a06e2cd2436477b7a06eddd': compiler.version=9
openssl/3.0.7-cb192d2@rutoken/experimental: 'b958caa7bf3c5d12ce630ace1dca6050750460ac': compiler.version=8
openssl/3.0.7-cb192d2@rutoken/experimental: '8030ccfecedce9534ec6270c27ed0c8e509879b6': build_type=Release, compiler.version=11.3
openssl/3.0.7-cb192d2@rutoken/experimental: 'c1485e746a617968790343b676fda7325cd621ae': build_type=Release, compiler.version=11.2
openssl/3.0.7-cb192d2@rutoken/experimental: '67013a1def2af33542ad14095334103725a3aa2b': build_type=Release, compiler.version=11.1
openssl/3.0.7-cb192d2@rutoken/experimental: '856e0d30187aeb0086f950f26e8a25190a2346c0': build_type=Release, compiler.version=10
openssl/3.0.7-cb192d2@rutoken/experimental: 'cf75bc26b4368738a060b1ef43b343b3e23e6dac': build_type=Release, compiler.version=9
openssl/3.0.7-cb192d2@rutoken/experimental: 'e838076049a737dc5b3c8878b7496f15abe73d78': build_type=Release, compiler.version=8
openssl/3.0.7-cb192d2@rutoken/experimental: Checking 13 compatible configurations:
openssl/3.0.7-cb192d2@rutoken/experimental: 'd26d4ad84e7d0d9ccde5c37a6514288f50e44efd': build_type=Release
openssl/3.0.7-cb192d2@rutoken/experimental: Main binary package '3a59bc00c13965211a71d35a72250fa70a09ccfb' missing. Using compatible package 'd26d4ad84e7d0d9ccde5c37a6514288f50e44efd'
pcsclite/1.9.5@rutoken/testing: Checking 13 compatible configurations:
pcsclite/1.9.5@rutoken/testing: '98981384b21184d62f24f26a66dd865b2fd3197c': build_type=Release
pcsclite/1.9.5@rutoken/testing: 'bebbe508853d2bdd2810cfe864433c55106d4592': compiler.version=11.3
pcsclite/1.9.5@rutoken/testing: 'ce749999665cb1e9ad3a0fad532f617efb7963fb': compiler.version=11.2
pcsclite/1.9.5@rutoken/testing: '1d247ccb365e7250c028c92d4707a3dc60190d0d': compiler.version=11.1
pcsclite/1.9.5@rutoken/testing: '1190665e21535be5ad003d4d50200ecea95b605e': compiler.version=10
pcsclite/1.9.5@rutoken/testing: '224a756f128ffa8f1f66c3d738b109de54b04764': compiler.version=9
pcsclite/1.9.5@rutoken/testing: '9a213b35a8582b079b675aeccf160eb34432ee79': compiler.version=8
pcsclite/1.9.5@rutoken/testing: '994fa52562bab39b4cc12155099ed0153ff13612': build_type=Release, compiler.version=11.3
pcsclite/1.9.5@rutoken/testing: 'e8d6525c180cc1a225de47c3e16ce35f9db9c6dc': build_type=Release, compiler.version=11.2
pcsclite/1.9.5@rutoken/testing: '5629499f747670e2210e2d68b4cb1efbbf276985': build_type=Release, compiler.version=11.1
pcsclite/1.9.5@rutoken/testing: 'bc8076832d87a9781b768d4832d0687717b41417': build_type=Release, compiler.version=10
pcsclite/1.9.5@rutoken/testing: '1a6fe6bf13de27fd3e21c89127fbbe271baf2681': build_type=Release, compiler.version=9
pcsclite/1.9.5@rutoken/testing: '60bb1855ba818f27d866e804b493b84c13f69f2d': build_type=Release, compiler.version=8
pcsclite/1.9.5@rutoken/testing: Checking 13 compatible configurations:
pcsclite/1.9.5@rutoken/testing: '98981384b21184d62f24f26a66dd865b2fd3197c': build_type=Release
pcsclite/1.9.5@rutoken/testing: 'bebbe508853d2bdd2810cfe864433c55106d4592': compiler.version=11.3
pcsclite/1.9.5@rutoken/testing: 'ce749999665cb1e9ad3a0fad532f617efb7963fb': compiler.version=11.2
pcsclite/1.9.5@rutoken/testing: '1d247ccb365e7250c028c92d4707a3dc60190d0d': compiler.version=11.1
pcsclite/1.9.5@rutoken/testing: '1190665e21535be5ad003d4d50200ecea95b605e': compiler.version=10
pcsclite/1.9.5@rutoken/testing: '224a756f128ffa8f1f66c3d738b109de54b04764': compiler.version=9
pcsclite/1.9.5@rutoken/testing: '9a213b35a8582b079b675aeccf160eb34432ee79': compiler.version=8
pcsclite/1.9.5@rutoken/testing: '994fa52562bab39b4cc12155099ed0153ff13612': build_type=Release, compiler.version=11.3
pcsclite/1.9.5@rutoken/testing: 'e8d6525c180cc1a225de47c3e16ce35f9db9c6dc': build_type=Release, compiler.version=11.2
pcsclite/1.9.5@rutoken/testing: '5629499f747670e2210e2d68b4cb1efbbf276985': build_type=Release, compiler.version=11.1
pcsclite/1.9.5@rutoken/testing: 'bc8076832d87a9781b768d4832d0687717b41417': build_type=Release, compiler.version=10
pcsclite/1.9.5@rutoken/testing: '1a6fe6bf13de27fd3e21c89127fbbe271baf2681': build_type=Release, compiler.version=9
pcsclite/1.9.5@rutoken/testing: '60bb1855ba818f27d866e804b493b84c13f69f2d': build_type=Release, compiler.version=8
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Checking 13 compatible configurations:
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '280af46ea68650b0df905d024c65f1b6ee455fe5': build_type=Release
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'bfc62b4d447120b9b4d441d539e36d2c4536ed6d': compiler.version=11.3
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '32620f2643a62f892e314aeb737d83b3124f5d5a': compiler.version=11.2
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'f0365d4feffde0ea7a85938009e457e5e94b61c1': compiler.version=11.1
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '18e202362cff25ab949b1b280185c53f1f3118a1': compiler.version=10
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'a9ae39abdee15e002579de03f881b6cbc75b1059': compiler.version=9
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '16816b0923d3abb21c0461363e21a1cbd1768847': compiler.version=8
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '3dd4b6a4c087c9c3686de50932f42a3ae6c2f1c5': build_type=Release, compiler.version=11.3
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'be42fa58baa687befcd429f1821409a8dcef0423': build_type=Release, compiler.version=11.2
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '085a46cd7ad31d1c5c9e65671e52f4d75db05d1b': build_type=Release, compiler.version=11.1
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'e2fa73dd3ebc38e37ffa0f22b835478f7b2b74f4': build_type=Release, compiler.version=10
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '7e5f49b00fa04da1b97e13dd68124ee51ae9ee64': build_type=Release, compiler.version=9
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '6938bb836441d28332649a9384984cd7dd8d2b1e': build_type=Release, compiler.version=8
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: Checking 13 compatible configurations:
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '280af46ea68650b0df905d024c65f1b6ee455fe5': build_type=Release
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'bfc62b4d447120b9b4d441d539e36d2c4536ed6d': compiler.version=11.3
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '32620f2643a62f892e314aeb737d83b3124f5d5a': compiler.version=11.2
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'f0365d4feffde0ea7a85938009e457e5e94b61c1': compiler.version=11.1
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '18e202362cff25ab949b1b280185c53f1f3118a1': compiler.version=10
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'a9ae39abdee15e002579de03f881b6cbc75b1059': compiler.version=9
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '16816b0923d3abb21c0461363e21a1cbd1768847': compiler.version=8
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '3dd4b6a4c087c9c3686de50932f42a3ae6c2f1c5': build_type=Release, compiler.version=11.3
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'be42fa58baa687befcd429f1821409a8dcef0423': build_type=Release, compiler.version=11.2
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '085a46cd7ad31d1c5c9e65671e52f4d75db05d1b': build_type=Release, compiler.version=11.1
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: 'e2fa73dd3ebc38e37ffa0f22b835478f7b2b74f4': build_type=Release, compiler.version=10
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '7e5f49b00fa04da1b97e13dd68124ee51ae9ee64': build_type=Release, compiler.version=9
pkcs11ecp/v2.7.1.0-2-g1b3b2602@rutoken/testing: '6938bb836441d28332649a9384984cd7dd8d2b1e': build_type=Release, compiler.version=8
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Checking 13 compatible configurations:
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: '426334e6c2a31bb34a190b48c3114ef5d7cb9140': build_type=Release
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental: Main binary package '090e2a50985ed8afc712ddce15c945dbe963b810' missing. Using compatible package '426334e6c2a31bb34a190b48c3114ef5d7cb9140'
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: Checking 1 compatible configurations:
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing: '50f5bb15f23a6e97a11679e5a1a884111d1fe8c3': build_type=Release
rtengine/v3.1.1-5-g5535964@rutoken/experimental: Checking 13 compatible configurations:
rtengine/v3.1.1-5-g5535964@rutoken/experimental: 'd1a1e4b7534bbfc447cde6cf7a56191051580ea1': build_type=Release
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '72cf280ae77105a4d385a492787320ef896ff4c6': compiler.version=11.3
rtengine/v3.1.1-5-g5535964@rutoken/experimental: 'c7d09cd7a47a84c615022523941606c9983b158a': compiler.version=11.2
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '674523e2f44d932345207260278c84a08be2906c': compiler.version=11.1
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '2584bc886a826de404259c0722f2428e98196b99': compiler.version=10
rtengine/v3.1.1-5-g5535964@rutoken/experimental: 'c6974c7b3c7f1c56547cef111d2f124f2cf325fc': compiler.version=9
rtengine/v3.1.1-5-g5535964@rutoken/experimental: 'f73abf5c01a21ae70868469db2dcaa25d44e3d0e': compiler.version=8
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '648aa58e4cb999a249ae5f0ea62fcec7fbd1683b': build_type=Release, compiler.version=11.3
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '9a5dd71a4fc1c5312e5160f1591043f07cdab177': build_type=Release, compiler.version=11.2
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '0c83e7eb12fdf56a3493a1337431248bb6ceda35': build_type=Release, compiler.version=11.1
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '70acc0dd4cadbed7bb6dd3e94dc390553b1bb2f6': build_type=Release, compiler.version=10
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '00ac2520d7d9afbc53d6e4a5a5775d64497de2c2': build_type=Release, compiler.version=9
rtengine/v3.1.1-5-g5535964@rutoken/experimental: '931121e3af148a6635680d0e5afbbda3a1905d4c': build_type=Release, compiler.version=8
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: Checking 13 compatible configurations:
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '64e987f38c32f37860afa13dbab2b0dcc0d0b6de': build_type=Release
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: 'a6fc06671067108ffc079ca856abd9e71b87e017': compiler.version=11.3
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '52543618d77f8a4cca90798f8700e3e9de1e0c58': compiler.version=11.2
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '56b943a16517ad6cb5ba73c348c29d43c2ed340e': compiler.version=11.1
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '9b234a35257b4a72248fd153067638e59d18b9a4': compiler.version=10
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '50fba9199f2b80de4f5ccf9ccb63ec1e24728cbe': compiler.version=9
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '26a63e57ca32c1bf9c625a969756a83be57ca854': compiler.version=8
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: 'a45feefe4fc940a752ab56ca87386252d84ebf02': build_type=Release, compiler.version=11.3
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: 'a2eee4c00b8f93c44ab9eac6746377f269d3585f': build_type=Release, compiler.version=11.2
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: 'fd03c70cdaad8783432e578a48df1bd8084df864': build_type=Release, compiler.version=11.1
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '4f423972e87e036c064f7180f341d8ef97a7d903': build_type=Release, compiler.version=10
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: 'd1e3822c9cfdb80fe9c668c813d35139ae32151f': build_type=Release, compiler.version=9
pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: '929912e34e25896683c0f067c67ba479c9697dd1': build_type=Release, compiler.version=8

======== Installing packages ========
ERROR: Missing binary: pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing:d9a2cef528d4b4d3a66e1724c68b253e57f11bb1

pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing: WARN: Can't find a 'pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing' package binary 'd9a2cef528d4b4d3a66e1724c68b253e57f11bb1' for the configuration:
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
libc=glibc
libc.version=2.12
os=Linux
[options]
country=ru
crypto=False
shared=True
sysformat=False
[requires]
cprocsp/5.0.11732@rutoken/experimental#75066d2cfa02a28f12b562ed4dae876d:f2d37cd7199b4fc0a2b397e64f1ae8247612826e
openssl/3.0.7-cb192d2@rutoken/experimental#c691c52600664b10b7477f3ca576ee9c:ce8dbdeab63ab1f0c7ace120ebfd3611aa5dd75f
pcsclite/1.9.5@rutoken/testing#a50e78deffa149d380cc0a88381cf124:222d94c573a4f4ae5eaee526bc5b3247046770f4
rtcrypt/v0.4.0-19-g84b054d@rutoken/experimental#1f4153c5c04cedc619e3faa2c9e025e8:426334e6c2a31bb34a190b48c3114ef5d7cb9140
rtengine/v3.1.1-5-g5535964@rutoken/experimental#6ab88a1007911bbc294a0ca014658a40:65abe93129f3465e8314430308210bd88e0dbd10
tokenchooser/2234642335593619d04b42b0030559b812c44b55@rutoken/testing#c1280aa905c0fe5ef843c05535fb05a1:f33bea03e0bf2394aee7ddebe87f0cc1795187e1
[python_requires]
rutokenconanfile/0.29.Z@rutoken/testing

ERROR: Missing prebuilt package for 'pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing'
Check the available packages using 'conan list pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing:* -r=remote'
or try to build locally from sources using the '--build=pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

Sorry, I can't write a small example, which reproduce my problem

Have you read the CONTRIBUTING guide?

memsharded commented 1 year ago

Hi @lo1ol

Lets try to first address and clarify some potential issues:

Regarding your questions:

Conan install works fine, but search for unnecessary packages (conan search for test_dependencies for test_dependencies) and has bad output(boost-mt is skipped but actually downloaded):

This is totally expected. The dependency graph in Conan 2.0 will always be complete, including all transitive dependencies of all requires, including test_requires and tool_requires. What will be skipped are the binaries, which are the heavy thing, but all recipes will always be fetched, as the graph needs to be completely defined. This was done because Conan 1.X had lot of issues exactly because it was pruning build_requires when not building from source.

The output seems correct wrt boost-mt, the recipe is downloaded, the binary is skipped:

Test requirements
    boost-mt/1.76.0-e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754 - Cache

Test requirements
boost-mt/1.76.0-e7488ca@rutoken/experimental#766....754:411....b24#272....4ae - Skip

but export-pkg looks for all compatible packages and can't find pkcs11ecp:

Yes, this certainly looks wrong. I can see in the first install:

pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79:435e75f0dbc065a8a7a66009de0c7c625aaf16cc#81d31254a330ea2f51c7f6f091dff66b - Download (rutoken)

And then in the export-pkg:

pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79 - Cache
...
ERROR: Missing binary: pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing:d9a2cef528d4b4d3a66e1724c68b253e57f11bb1

The recipe revision is the same, but the package_id is different. I have been looking at the code and the output, but I have no idea why. It would be necessary to reduce the problem and have something that we could reproduce in order to be able to investigate this.

It looks that other package binaries are missing as well, for rtcrypt and other packages. I'd say that it could be something related to the dynamic definition of options. Doing a conan list pkcs11ecp/v2.7.1.0-4-gaa62c24b@rutoken/testing#c0a84b2701d504989b1bbf53a4663f79:* to list the current existing binaries could also help to see where the differences are.

lo1ol commented 1 year ago

conan version 2.0.1

It reads unexpected that you depend on openssl as a test_requires, but not as a normal requires. This should mean that only the test executables depends on openssl, but your final library, the one packaged doesn't have any kind of dependency, not visible or invisible to openssl.

Yes. Openssl is used only for tests at rtengine (it's openssl engine:)). Users of rtengine may use it with self version of openssl.

I think you might be using too much the visible trait. This is expected to be used only to address some kind of issues like complete isolation, but it shouldn't be necessary in most "shared lib links static lib", as this is already automatically managed.

openssl and rtengine not visible for pkcs11ecp shared lib, because shared version of pkcs11ecp contains openssl and rtengine symbols inside (for internal cryptography). Users of shared version pkcs11ecp doesn't need to think about that. It's needed because not all target distro contains necessary version of openssl

The dependency graph in Conan 2.0 will always be complete, including all transitive dependencies of all requires, including test_requires and tool_requires.

But in my case it's means that after a lot of iteration (where rtengine and pkcs11ecp would be depend on each other), conan would search for binary packages of all previous versions of rtengine and pkcs11ecp. It will consume a lot of time!

The output seems correct wrt boost-mt, the recipe is downloaded, the binary is skipped:

The error not at there:) rtengine search for boost-mt specified at current conanfile (which actually downloaded according to log) and on boost-mt specified at previous version of rtengine (which actually skipped). But at the log only skipped is specified. I think that correct log should contain something like that:

    boost-mt/1.76.0-e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754:4115bd5b2ad61fe73c9251212dce1dca55df9b24#27200b190241c7a40892d7b861ab44ae - Skip
e7488ca@rutoken/experimental#766f131873c73bc3cde079bff2bf1754:4115bd5b2ad61fe73c9251212dce1dca55df9b24#27200b190241c7a40892d7b861ab44ae - Download (rutoken)

or doesn't look for boost-mt transited from previous version of rtengine at all.

. It would be necessary to reduce the problem and have something that we could reproduce in order to be able to investigate this.

Ok. I try this

lo1ol commented 1 year ago

I found smallest example to reproduce the problem:

pkgA.py

from conan import ConanFile

class PcscLiteConanfile(ConanFile):
    name="pkg_a"
    version="0.0.1"

    settings = "build_type"

    def compatibility(self):
        if self.settings.build_type == "Debug":
            return [{"settings": [("build_type", "Release")]}]

pkgB.py:

from conan import ConanFile

class RtPkcs11EcpConanfile(ConanFile):
    name = "pkg_b"
    version = "0.0.1"

    settings = "build_type"
    options = {"shared": [True, False]} # this is essential
    default_options = {"shared": True }

    def build_requirements(self):
        self.test_requires("pkg_a/0.0.1")

pkgC.py

from conan import ConanFile

class RtengineConanfile(ConanFile):
    name = "pkg_c"
    version = "0.0.1"

    settings = "build_type"

    def build_requirements(self):
        self.test_requires("pkg_b/0.0.1")
#!/bin/bash -eu

conan remove -c "*"
conan remove -r rutoken -c "pkg_a" |:

echo "install pkgA"
conan install pkgA.py
echo "export-pkg pkgA"
conan export-pkg pkgA.py
conan upload -r rutoken -c "pkg_a"

echo "install pkgB"
conan install -s build_type=Debug pkgB.py
echo "export-pkg pkgB"
conan export-pkg -s build_type=Debug pkgB.py

conan remove -c "pkg_a" # This is essential command

echo "install pkgC"
conan install -s build_type=Debug pkgC.py
echo "export-pkg pkgC"
conan export-pkg -s build_type=Debug pkgC.py

If remove shared options from pkgB.py (or replace it to another option. for example share1) everything is work.

also if don't remove pkg_b from local cache -- everything works too

lo1ol commented 1 year ago

log:

./test.sh
install pkgA

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

======== Computing dependency graph ========
Graph root
    pkgA.py (pkg_a/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgA.py

======== Computing necessary packages ========

======== Installing packages ========

======== Finalizing install (deploy, generators) ========
pkgA.py (pkg_a/0.0.1): Generating aggregated env files
pkgA.py (pkg_a/0.0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully
export-pkg pkgA

======== Exporting recipe to the cache ========
pkg_a/0.0.1: Exporting package recipe: /home/lo1ol/Documents/work/test-conan/pkgA.py
pkg_a/0.0.1: Copied 1 '.py' file: conanfile.py
pkg_a/0.0.1: Exported to cache folder: /home/lo1ol/.conan2/p/pkg_ab631e374f968e/e
pkg_a/0.0.1: Exported: pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e (2023-03-14 14:12:13 UTC)

======== Computing dependency graph ========
Graph root
    pkgA.py (pkg_a/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgA.py

======== Computing necessary packages ========

======== Installing packages ========
pkgA.py (pkg_a/0.0.1): Generating aggregated env files
pkgA.py (pkg_a/0.0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']

======== Exporting recipe and package to the cache ========
pkgA.py (pkg_a/0.0.1): Exporting binary from user folder to Conan cache
pkgA.py (pkg_a/0.0.1): Packaging to efa83b160a55b033c4ea706ddb980cd708e3ba1b
pkgA.py (pkg_a/0.0.1): Generating the package
pkgA.py (pkg_a/0.0.1): Temporary package folder /home/lo1ol/.conan2/p/t/pkg_a69a64e75175a7/p
pkgA.py (pkg_a/0.0.1): package(): WARN: No files in this package!
pkgA.py (pkg_a/0.0.1): Created package revision ccf1fabeefeb701e3c87002e9d7adefb
pkgA.py (pkg_a/0.0.1): Package 'efa83b160a55b033c4ea706ddb980cd708e3ba1b' created
pkgA.py (pkg_a/0.0.1): Full package reference: pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e:efa83b160a55b033c4ea706ddb980cd708e3ba1b#ccf1fabeefeb701e3c87002e9d7adefb
pkgA.py (pkg_a/0.0.1): Package folder /home/lo1ol/.conan2/p/pkg_a9bd4b05a8380f/p
pkgA.py (pkg_a/0.0.1): Exported package binary
Checking which revisions exist in the remote server
Preparing artifacts to upload
Uploading artifacts
Uploading recipe 'pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e'
Uploading package 'pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e:efa83b160a55b033c4ea706ddb980cd708e3ba1b#ccf1fabeefeb701e3c87002e9d7adefb'
install pkgB

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

======== Computing dependency graph ========
Graph root
    pkgB.py (pkg_b/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgB.py
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e - Cache

======== Computing necessary packages ========
pkg_a/0.0.1: Checking 1 compatible configurations:
pkg_a/0.0.1: 'efa83b160a55b033c4ea706ddb980cd708e3ba1b': build_type=Release
pkg_a/0.0.1: Main binary package '9e186f6d94c008b544af1569d1a6368d8339efc5' missing. Using compatible package 'efa83b160a55b033c4ea706ddb980cd708e3ba1b'
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e:efa83b160a55b033c4ea706ddb980cd708e3ba1b#ccf1fabeefeb701e3c87002e9d7adefb - Cache

======== Installing packages ========
pkg_a/0.0.1: Already installed!

======== Finalizing install (deploy, generators) ========
pkgB.py (pkg_b/0.0.1): Generating aggregated env files
pkgB.py (pkg_b/0.0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully
export-pkg pkgB

======== Exporting recipe to the cache ========
pkg_b/0.0.1: Exporting package recipe: /home/lo1ol/Documents/work/test-conan/pkgB.py
pkg_b/0.0.1: Copied 1 '.py' file: conanfile.py
pkg_b/0.0.1: Exported to cache folder: /home/lo1ol/.conan2/p/pkg_b10d7d875a8e0e/e
pkg_b/0.0.1: Exported: pkg_b/0.0.1#a16496bc9a23c07d8239ff4f1c163d91 (2023-03-14 14:12:22 UTC)

======== Computing dependency graph ========
Graph root
    pkgB.py (pkg_b/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgB.py
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e - Cache

======== Computing necessary packages ========
pkg_a/0.0.1: Checking 1 compatible configurations:
pkg_a/0.0.1: 'efa83b160a55b033c4ea706ddb980cd708e3ba1b': build_type=Release
pkg_a/0.0.1: Main binary package '9e186f6d94c008b544af1569d1a6368d8339efc5' missing. Using compatible package 'efa83b160a55b033c4ea706ddb980cd708e3ba1b'

======== Installing packages ========
pkg_a/0.0.1: Already installed!
pkgB.py (pkg_b/0.0.1): Generating aggregated env files
pkgB.py (pkg_b/0.0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']

======== Exporting recipe and package to the cache ========
pkgB.py (pkg_b/0.0.1): Exporting binary from user folder to Conan cache
pkgB.py (pkg_b/0.0.1): Packaging to 6d8ab77a791939f904862f0f55d62b5d637d11e8
pkgB.py (pkg_b/0.0.1): Generating the package
pkgB.py (pkg_b/0.0.1): Temporary package folder /home/lo1ol/.conan2/p/t/pkg_b7ec656f907f27/p
pkgB.py (pkg_b/0.0.1): package(): WARN: No files in this package!
pkgB.py (pkg_b/0.0.1): Created package revision 4ac1919bee8871c1d9bbdca325445088
pkgB.py (pkg_b/0.0.1): Package '6d8ab77a791939f904862f0f55d62b5d637d11e8' created
pkgB.py (pkg_b/0.0.1): Full package reference: pkg_b/0.0.1#a16496bc9a23c07d8239ff4f1c163d91:6d8ab77a791939f904862f0f55d62b5d637d11e8#4ac1919bee8871c1d9bbdca325445088
pkgB.py (pkg_b/0.0.1): Package folder /home/lo1ol/.conan2/p/pkg_bff02126c4a346/p
pkgB.py (pkg_b/0.0.1): Exported package binary
install pkgC

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Debug
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

Profile build:
[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=12
os=Linux

======== Computing dependency graph ========
pkg_a/0.0.1: Not found in local cache, looking in remotes...
pkg_a/0.0.1: Checking remote: conancenter
pkg_a/0.0.1: Checking remote: rutoken
pkg_a/0.0.1: Downloaded recipe revision 6c09c6650467d7ab169311fca3aefb1e
Graph root
    pkgC.py (pkg_c/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgC.py
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e - Downloaded (rutoken)
    pkg_b/0.0.1#a16496bc9a23c07d8239ff4f1c163d91 - Cache

======== Computing necessary packages ========
pkg_a/0.0.1: Checking 1 compatible configurations:
pkg_a/0.0.1: 'efa83b160a55b033c4ea706ddb980cd708e3ba1b': build_type=Release
pkg_a/0.0.1: Main binary package '9e186f6d94c008b544af1569d1a6368d8339efc5' missing. Using compatible package 'efa83b160a55b033c4ea706ddb980cd708e3ba1b'
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e:efa83b160a55b033c4ea706ddb980cd708e3ba1b#ccf1fabeefeb701e3c87002e9d7adefb - Skip
    pkg_b/0.0.1#a16496bc9a23c07d8239ff4f1c163d91:6d8ab77a791939f904862f0f55d62b5d637d11e8#4ac1919bee8871c1d9bbdca325445088 - Cache

======== Installing packages ========
pkg_b/0.0.1: Already installed!

======== Finalizing install (deploy, generators) ========
pkgC.py (pkg_c/0.0.1): Generating aggregated env files
pkgC.py (pkg_c/0.0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished succesfully
export-pkg pkgC

======== Exporting recipe to the cache ========
pkg_c/0.0.1: Exporting package recipe: /home/lo1ol/Documents/work/test-conan/pkgC.py
pkg_c/0.0.1: Copied 1 '.py' file: conanfile.py
pkg_c/0.0.1: Exported to cache folder: /home/lo1ol/.conan2/p/pkg_cc6900ae14aefb/e
pkg_c/0.0.1: Exported: pkg_c/0.0.1#39810596b16e86dea9373ccdc6694416 (2023-03-14 14:12:31 UTC)

======== Computing dependency graph ========
Graph root
    pkgC.py (pkg_c/0.0.1): /home/lo1ol/Documents/work/test-conan/pkgC.py
Test requirements
    pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e - Cache
    pkg_b/0.0.1#a16496bc9a23c07d8239ff4f1c163d91 - Cache

======== Computing necessary packages ========
pkg_a/0.0.1: Checking 1 compatible configurations:
pkg_a/0.0.1: 'efa83b160a55b033c4ea706ddb980cd708e3ba1b': build_type=Release

======== Installing packages ========
ERROR: Missing binary: pkg_b/0.0.1:b0dc27396d16a366969b29893948bc907ba311b2

pkg_b/0.0.1: WARN: Can't find a 'pkg_b/0.0.1' package binary 'b0dc27396d16a366969b29893948bc907ba311b2' for the configuration:
[settings]
build_type=Debug
[options]
shared=True
[requires]
pkg_a/0.0.1#6c09c6650467d7ab169311fca3aefb1e:9e186f6d94c008b544af1569d1a6368d8339efc5

ERROR: Missing prebuilt package for 'pkg_b/0.0.1'
Check the available packages using 'conan list pkg_b/0.0.1:* -r=remote'
or try to build locally from sources using the '--build=pkg_b/0.0.1' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'
lo1ol commented 1 year ago

I think that the root of the problem is that export-pkg doesn't look compatible packages at remotes. As you say list of packages are used for hash computation, test_packages is used too, but may be uninstalled. We can't compute correct hash if doesn't know used compatible binary package.

This patch resolve this:

diff --git a/conan/cli/commands/export_pkg.py b/conan/cli/commands/export_pkg.py
index 0e6af84..d71f6bc 100644
--- a/conan/cli/commands/export_pkg.py
+++ b/conan/cli/commands/export_pkg.py
@@ -19,11 +19,6 @@ def export_pkg(conan_api, parser, *args):
     """
     Create a package directly from pre-compiled binaries.
     """
+    group = parser.add_mutually_exclusive_group()
+    group.add_argument("-r", "--remote", action="append", default=None,
+                       help='Look in the specified remote or remotes server')
+    group.add_argument("-nr", "--no-remote", action="store_true",
+                       help='Do not use remote, resolve exclusively in the cache')
     parser.add_argument("path", help="Path to a folder containing a recipe (conanfile.py)")
     parser.add_argument("-of", "--output-folder",
                         help='The root output folder for generated and build files')
@@ -43,7 +38,6 @@ def export_pkg(conan_api, parser, *args):
     test_conanfile_path = _get_test_conanfile_path(args.test_folder, path)
     lockfile = conan_api.lockfile.get_lockfile(lockfile=args.lockfile, conanfile_path=path,
                                                cwd=cwd, partial=args.lockfile_partial)
+    remotes = conan_api.remotes.list(args.remote) if not args.no_remote else []
     profile_host, profile_build = conan_api.profiles.get_profiles_from_args(args)

     ref, conanfile = conan_api.export.export(path=path, name=args.name, version=args.version,
@@ -58,12 +52,12 @@ def export_pkg(conan_api, parser, *args):
                                                      ref.name, ref.version, ref.user, ref.channel,
                                                      profile_host=profile_host,
                                                      profile_build=profile_build,
-                                                     lockfile=lockfile, remotes=None, update=None,
+                                                     lockfile=lockfile, remotes=remotes, update=None,
                                                      is_build_require=args.build_require)

     print_graph_basic(deps_graph)
     deps_graph.report_graph_error()
-    conan_api.graph.analyze_binaries(deps_graph, build_mode=[ref.name], lockfile=lockfile, remotes=remotes)
+    conan_api.graph.analyze_binaries(deps_graph, build_mode=[ref.name], lockfile=lockfile)
     deps_graph.report_graph_error()

     root_node = deps_graph.root
@@ -71,7 +65,7 @@ def export_pkg(conan_api, parser, *args):

     # It is necessary to install binaries, in case there are build_requires necessary to export
     # But they should be local, if this was built here
-    conan_api.install.install_binaries(deps_graph=deps_graph, remotes=None)
+    conan_api.install.install_binaries(deps_graph=deps_graph, remotes=remotes)
     source_folder = os.path.dirname(path)
     output_folder = make_abs_path(args.output_folder, cwd) if args.output_folder else None
     conan_api.install.install_consumer(deps_graph=deps_graph, source_folder=source_folder,
@@ -86,7 +80,7 @@ def export_pkg(conan_api, parser, *args):
     if test_conanfile_path:
         from conan.cli.commands.test import run_test
         deps_graph = run_test(conan_api, test_conanfile_path, ref, profile_host, profile_build,
-                              remotes=None, lockfile=lockfile, update=False, build_modes=None)
+                              remotes=remotes, lockfile=lockfile, update=False, build_modes=None)

But my ask still unresolved. How can I avoid potential huge graph in the future when packages reference to each other as test_require?

May be you add option to explicitly exclude package from graph for consumers? We use straight versioning at out projects, so our dependency graph can't be changed after a while. It may be changed only if new compatible package appears, but in most cases it doesn't matter for test packages (as I think) and adding special option to accept this and exclude package from consumer's graph will be useful.

lo1ol commented 1 year ago

The main idea I would like to say -- "Test packages shouldn't affect on the target binary package in 99% cases"

lo1ol commented 1 year ago

It's my draft suggestion

diff --git a/conans/client/graph/compute_pid.py b/conans/client/graph/compute_pid.py
index 61a80f1d8..d45fca6f4 100644
--- a/conans/client/graph/compute_pid.py
+++ b/conans/client/graph/compute_pid.py
@@ -25,6 +25,9 @@ def compute_package_id(node, new_config):
     data = OrderedDict()
     build_data = OrderedDict()
     for require, transitive in node.transitive_deps.items():
+        if require.exclude_from_consumer_graph:
+            continue
+
         dep_node = transitive.node
         require.deduce_package_id_mode(conanfile.package_type, dep_node,
                                        non_embed_mode, embed_mode, build_mode, unknown_mode)
diff --git a/conans/client/graph/graph_builder.py b/conans/client/graph/graph_builder.py
index 079841ad3..c0a148832 100644
--- a/conans/client/graph/graph_builder.py
+++ b/conans/client/graph/graph_builder.py
@@ -46,6 +46,10 @@ class DepsGraphBuilder(object):
             while open_requires:
                 # Fetch the first waiting to be expanded (depth-first)
                 (require, node) = open_requires.popleft()
+
+                if node != root_node and require.exclude_from_consumer_graph:
+                    continue
+
                 if require.override:
                     continue
                 new_node = self._expand_require(require, node, dep_graph, profile_host,
@@ -55,6 +59,7 @@ class DepsGraphBuilder(object):
                     open_requires.extendleft((r, new_node)
                                              for r in reversed(new_node.conanfile.requires.values()))
             self._remove_overrides(dep_graph)
+            self._remove_excluded_for_consume(dep_graph)
             check_graph_provides(dep_graph)
             self._compute_test_package_deps(dep_graph)
         except GraphError as e:
@@ -293,6 +298,13 @@ class DepsGraphBuilder(object):
             for r in to_remove:
                 node.transitive_deps.pop(r)

+    @staticmethod
+    def _remove_excluded_for_consume(dep_graph):
+        for node in dep_graph.nodes[1:]:
+            to_remove = [r for r in node.transitive_deps if r.exclude_from_consumer_graph]
+            for r in to_remove:
+                node.transitive_deps.pop(r)
+
     @staticmethod
     def _compute_test_package_deps(graph):
         """ compute and tag the graph nodes that belong exclusively to test_package
diff --git a/conans/model/requires.py b/conans/model/requires.py
index 28fb742bf..d44806887 100644
--- a/conans/model/requires.py
+++ b/conans/model/requires.py
@@ -11,7 +11,7 @@ class Requirement:
     """
     def __init__(self, ref, *, headers=None, libs=None, build=False, run=None, visible=None,
                  transitive_headers=None, transitive_libs=None, test=None, package_id_mode=None,
-                 force=None, override=None, direct=None, options=None):
+                 force=None, override=None, direct=None, options=None, exclude_from_consumer_graph=None):
         # * prevents the usage of more positional parameters, always ref + **kwargs
         # By default this is a generic library requirement
         self.ref = ref
@@ -27,6 +27,7 @@ class Requirement:
         self._force = force
         self._override = override
         self._direct = direct
+        self._exclude_from_consumer_graph = exclude_from_consumer_graph
         self.options = options

     @property
@@ -93,6 +94,14 @@ class Requirement:
     def direct(self, value):
         self._direct = value

+    @property
+    def exclude_from_consumer_graph(self):
+        return self._default_if_none(self._exclude_from_consumer_graph, False)
+
+    @exclude_from_consumer_graph.setter
+    def exclude_from_consumer_graph(self, value):
+        self._exclude_from_consumer_graph = value
+
     @property
     def build(self):
         return self._build
@@ -526,7 +535,7 @@ class Requirements:
         # libs = True => We need to link with it
         # headers = True => We need to include it
         req = Requirement(ref, headers=True, libs=True, build=False, run=run, visible=False,
-                          test=True, package_id_mode=None, options=options)
+                          test=True, exclude_from_consumer_graph=True, package_id_mode=None, options=options)
         if self._requires.get(req):
             raise ConanException("Duplicated requirement: {}".format(ref))
         self._requires[req] = req

This patch fix the problem of missing binary test package too.

lo1ol commented 1 year ago

@memsharded what do you think about this?:)

lo1ol commented 1 year ago

I find out why shared option affects on search. When conan recognizes package as shared library, then package_id_mode=full_mode. If remove shared option, conan search packages by semver_mode.

lo1ol commented 1 year ago

It seems that using of full_recipe_mode for test packages may partly resolve my problem. But I still want to remove test packages from graph completely

lo1ol commented 1 year ago

Oh. I notice that you add ability to specify remotes for export-pkg. Thanks! This partially resolve my problem

memsharded commented 1 year ago

Hi @lo1ol

Sorry for the delay, too busy here. Yes, the remotes management for export-pkg was recently added.

Regarding the removal of things from the graph, we really don't want to go that way, it was really problematic and painful in 1.X, affecting things like reproducibility, lockfiles, CI, etc. It really shouldn't be an issue, thanks very much for providing the above repro case, I will convert it to a test in the codebase to properly check this.

lo1ol commented 1 year ago

But how do you will be resolve the problem of huge graph for packages, which contains test_requires to each other? After 10 version updates, conan would search for 20 old packages.

And it's best case scenario. If one of this packages would contain another deps, then this deps would be searched too!

lo1ol commented 1 year ago

I see only one solution which doesn't break compatibility and boost search for this requires. If non visible require (or its dependency) has package_id_mode, which doesn't include package_id and recipe_revision, then conan may skip searching of binary package for this kind of deps

memsharded commented 1 year ago

But how do you will be resolve the problem of huge graph for packages, which contains test_requires to each other? After 10 version updates, conan would search for 20 old packages.

And it's best case scenario. If one of this packages would contain another deps, then this deps would be searched too!

Even for 20 packages, it is downloading only the recipes, not the packages. It means adding a few seconds to the build, which is typically negligible compared with all the other things.

Also, the recommendation is that they shouldn't diverge that much, and the usage of test frameworks libraries also keeps maintained, upgrading to latest versions when possible, using version-ranges to automate the usage of the latest in the valid range, etc, etc, so ideally only 1 or a very few different versions of the test framework.

then conan may skip searching of binary package for this kind of deps

yes, this is the expected behavior, and this is what I need to double check. The expected behavior is that these dependencies are "Skipped" that means that the binary is not retrieved.

lo1ol commented 1 year ago

No. It doesn't just download only recipes. It may search for compatible packages too. And sometimes it may take a while.

lo1ol commented 1 year ago

In my case looking for compatible packages sometime is most time consuming process

memsharded commented 1 year ago

It may search for compatible packages too. And sometimes it may take a while.

That is a different issue. We need to add a new endpoint in the server to optimize this, but this will take more time, because server side modifications are always more challenging.

In any case, if you want to disable that and you are providing your own compatibility.py plugin implementation, I think we might give visibility over the test attribute to conanfile so you could inhibit the extra checks for test requires with 1 line

lo1ol commented 1 year ago

Here is example when searching for skipped packages take a lot of time:

$ conan install -pr android-armv7a-clang . -s build_type=Debug &| python -c 'import sys,time;sys.stderr.write("".join(( " ".join((time.strftime("[%Y-%m-%d %H:%M:%S]", 
time.localtime()), line)) for line in sys.stdin )))'
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] ======== Input profiles ========
[2023-03-16 15:35:31] Profile host:
[2023-03-16 15:35:31] [settings]
[2023-03-16 15:35:31] arch=armv7a
[2023-03-16 15:35:31] build_type=Debug
[2023-03-16 15:35:31] compiler=clang
[2023-03-16 15:35:31] compiler.cppstd=17
[2023-03-16 15:35:31] compiler.libcxx=c++_static
[2023-03-16 15:35:31] compiler.version=9
[2023-03-16 15:35:31] os=Android
[2023-03-16 15:35:31] os.api_level=21
[2023-03-16 15:35:31] [conf]
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] Profile build:
[2023-03-16 15:35:31] [settings]
[2023-03-16 15:35:31] arch=x86_64
[2023-03-16 15:35:31] build_type=Release
[2023-03-16 15:35:31] compiler=gcc
[2023-03-16 15:35:31] compiler.cppstd=gnu17
[2023-03-16 15:35:31] compiler.libcxx=libstdc++11
[2023-03-16 15:35:31] compiler.version=12
[2023-03-16 15:35:31] os=Linux
[2023-03-16 15:35:31] [conf]
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] ======== Computing dependency graph ========
[2023-03-16 15:35:31] Graph root
[2023-03-16 15:35:31]     conanfile.py: /home/lo1ol/Documents/work/rtengine/conanfile.py
[2023-03-16 15:35:31] Requirements
[2023-03-16 15:35:31]     rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental#da2eb5b905fd4bae6ae02f8014871b17 - Cache
[2023-03-16 15:35:31] Test requirements
[2023-03-16 15:35:31]     boost-mt/1.76.0-95c42d0@rutoken/experimental#4818bcb8ce059af9ea6874c44f379a16 - Cache
[2023-03-16 15:35:31]     openssl/3.0.7-ad46d99@rutoken/experimental#4a4d2bc5ff1b06ac6b243ec84e848738 - Cache
[2023-03-16 15:35:31]     pcsc-android/v1.10.0-17-g5be812f8@rutoken/testing#51831e845a7d45e762b4283d66cd940e - Cache
[2023-03-16 15:35:31]     pkcs11ecp/v2.7.1.0-5-geaaa9899@rutoken/testing#af35daab03120ffdc6151d93f023b531 - Cache
[2023-03-16 15:35:31]     pkcs11ecp/v2.7.1.0-7-gbd5245fc@rutoken/testing#bb0ff9e7ca791358067132c8684c9089 - Cache
[2023-03-16 15:35:31]     rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental#da2eb5b905fd4bae6ae02f8014871b17 - Cache
[2023-03-16 15:35:31]     rtengine/v3.1.1-5-g002edde@rutoken/experimental#a389b34fba927322bb3ad85929d176d5 - Cache
[2023-03-16 15:35:31]     rtservice/v1.10.0-17-g5be812f8@rutoken/testing#6f3c550dc0746566e15b3ef941ea782a - Cache
[2023-03-16 15:35:31]     tokenchooser/ec349279f9f5397038b0b54e8c9fb0b4ace3c9c2@rutoken/experimental#0bc238aad574f1f85a0a233b6176e96d - Cache
[2023-03-16 15:35:31] Python requires
[2023-03-16 15:35:31]     rutokenconanfile/0.30@rutoken/testing#852b7e76cc98a0990b10ad60853efc04 - Cache
[2023-03-16 15:35:31]     rutokenconanfile/0.33@rutoken/testing#4c41e2d3c090c91df961cdc3bddd9838 - Cache
[2023-03-16 15:35:31] 
[2023-03-16 15:35:31] ======== Computing necessary packages ========
[2023-03-16 15:35:33] boost-mt/1.76.0-95c42d0@rutoken/experimental: Checking 5 compatible configurations:
[2023-03-16 15:35:33] boost-mt/1.76.0-95c42d0@rutoken/experimental: '8d9a1536823b12f575f3adbd5a98fef59dec88d4': build_type=Release
[2023-03-16 15:35:33] boost-mt/1.76.0-95c42d0@rutoken/experimental: Main binary package 'bf288a9298e97d570dbf7e754035d144df34c41f' missing. Using compatible package '8d9a1536823b12f575f3adbd5a98fef59dec88d4'
[2023-03-16 15:35:33] openssl/3.0.7-ad46d99@rutoken/experimental: Checking 1 compatible configurations:
[2023-03-16 15:35:33] openssl/3.0.7-ad46d99@rutoken/experimental: '22ab0e7653c4b940d6c07771ac4ca7173608a38e': build_type=Release
[2023-03-16 15:35:34] openssl/3.0.7-ad46d99@rutoken/experimental: Main binary package 'd4f4136e6eaa46a182b600d2686d82268890782d' missing. Using compatible package '22ab0e7653c4b940d6c07771ac4ca7173608a38e'
[2023-03-16 15:35:35] openssl/3.0.7-ad46d99@rutoken/experimental: Checking 1 compatible configurations:
[2023-03-16 15:35:35] openssl/3.0.7-ad46d99@rutoken/experimental: '2ac8eebcb01d5a3aa20d2cec95878f88e56542f3': build_type=Release
[2023-03-16 15:35:35] openssl/3.0.7-ad46d99@rutoken/experimental: Main binary package 'c80cf8bbe4be936181bfcd4dd2b92ace75e47a96' missing. Using compatible package '2ac8eebcb01d5a3aa20d2cec95878f88e56542f3'
[2023-03-16 15:35:37] rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental: Checking 1 compatible configurations:
[2023-03-16 15:35:37] rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental: '51fcf6129e478795d3696e85f7dd554dd4ec7e51': build_type=Release
[2023-03-16 15:35:37] rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental: Main binary package 'fa4ffb60c847e6205560ca42785c2186cd9f904c' missing. Using compatible package '51fcf6129e478795d3696e85f7dd554dd4ec7e51'
[2023-03-16 15:35:38] rtengine/v3.1.1-5-g002edde@rutoken/experimental: Checking 1 compatible configurations:
[2023-03-16 15:35:38] rtengine/v3.1.1-5-g002edde@rutoken/experimental: '2f1a76026c88b3ccd772f06e220dc1bb4133b1df': build_type=Release
[2023-03-16 15:35:39] rtengine/v3.1.1-5-g002edde@rutoken/experimental: Main binary package '36cc47c2229ef7f3cb3417abd35b0cd1091cc85c' missing. Using compatible package '2f1a76026c88b3ccd772f06e220dc1bb4133b1df'
[2023-03-16 15:35:39] tokenchooser/ec349279f9f5397038b0b54e8c9fb0b4ace3c9c2@rutoken/experimental: Checking 1 compatible configurations:
[2023-03-16 15:35:39] tokenchooser/ec349279f9f5397038b0b54e8c9fb0b4ace3c9c2@rutoken/experimental: '4284ea20829a1cb2e7033ff724481be706b11636': build_type=Release
[2023-03-16 15:35:40] tokenchooser/ec349279f9f5397038b0b54e8c9fb0b4ace3c9c2@rutoken/experimental: Main binary package 'bdf439adae7825ea248a59ad90752448f333b0d6' missing. Using compatible package '4284ea20829a1cb2e7033ff724481be706b11636'
[2023-03-16 15:35:40] Requirements
[2023-03-16 15:35:40]     rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental#da2eb5b905fd4bae6ae02f8014871b17:51fcf6129e478795d3696e85f7dd554dd4ec7e51#8a3f7a012dd4e69fd731ba3105b9029a - Cache
[2023-03-16 15:35:40] Test requirements
[2023-03-16 15:35:40]     boost-mt/1.76.0-95c42d0@rutoken/experimental#4818bcb8ce059af9ea6874c44f379a16:8d9a1536823b12f575f3adbd5a98fef59dec88d4#da67258a5d3218ccf8349bdbf3fdd90b - Skip
[2023-03-16 15:35:40]     openssl/3.0.7-ad46d99@rutoken/experimental#4a4d2bc5ff1b06ac6b243ec84e848738:22ab0e7653c4b940d6c07771ac4ca7173608a38e#b9d79e3672c80e1fcb97ead5f1758bcd - Skip
[2023-03-16 15:35:40]     openssl/3.0.7-ad46d99@rutoken/experimental#4a4d2bc5ff1b06ac6b243ec84e848738:2ac8eebcb01d5a3aa20d2cec95878f88e56542f3#39eb7aea343bc8468ab134d555c0263b - Cache
[2023-03-16 15:35:40]     pcsc-android/v1.10.0-17-g5be812f8@rutoken/testing#51831e845a7d45e762b4283d66cd940e:79763d29f672ab96687237a7c7fd3a29d67ee27f#f82a7d07628a4d1cb0d62756babbe96c - Skip
[2023-03-16 15:35:40]     pkcs11ecp/v2.7.1.0-5-geaaa9899@rutoken/testing#af35daab03120ffdc6151d93f023b531:9b0e688eb1ee4bb4a8892a1563c6459cb3aa1e1d#2819fa2e36a60d952c47b1782dbf0049 - Skip
[2023-03-16 15:35:40]     pkcs11ecp/v2.7.1.0-7-gbd5245fc@rutoken/testing#bb0ff9e7ca791358067132c8684c9089:e85a8aec85d0d22fabfaf62d4575020a5179ddfd#ee2dde9e07cfbf24f4bb3a61f32d65e2 - Cache
[2023-03-16 15:35:40]     rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental#da2eb5b905fd4bae6ae02f8014871b17:51fcf6129e478795d3696e85f7dd554dd4ec7e51#8a3f7a012dd4e69fd731ba3105b9029a - Skip
[2023-03-16 15:35:40]     rtengine/v3.1.1-5-g002edde@rutoken/experimental#a389b34fba927322bb3ad85929d176d5:2f1a76026c88b3ccd772f06e220dc1bb4133b1df#0c66485d728df5728bb51a8ac68d449e - Skip
[2023-03-16 15:35:40]     rtservice/v1.10.0-17-g5be812f8@rutoken/testing#6f3c550dc0746566e15b3ef941ea782a:2bbf4b69c14478ebed531ee72d9de17567973269#ec0681d0384e89e2847d5fb9a1401316 - Skip
[2023-03-16 15:35:40]     tokenchooser/ec349279f9f5397038b0b54e8c9fb0b4ace3c9c2@rutoken/experimental#0bc238aad574f1f85a0a233b6176e96d:4284ea20829a1cb2e7033ff724481be706b11636#498ae96bf188dbadc9541d4abbd36bbf - Skip
[2023-03-16 15:35:40] 
[2023-03-16 15:35:40] ======== Installing packages ========
[2023-03-16 15:35:40] boost-mt/1.76.0-95c42d0@rutoken/experimental: Already installed! (1 of 4)
[2023-03-16 15:35:40] openssl/3.0.7-ad46d99@rutoken/experimental: Already installed! (2 of 4)
[2023-03-16 15:35:40] rtcrypt/v0.4.0-19-g6d9af8f@rutoken/experimental: Already installed! (3 of 4)
[2023-03-16 15:35:40] pkcs11ecp/v2.7.1.0-7-gbd5245fc@rutoken/testing: Already installed! (4 of 4)
[2023-03-16 15:35:40] 
[2023-03-16 15:35:40] ======== Finalizing install (deploy, generators) ========
[2023-03-16 15:35:40] conanfile.py: Calling generate()
[2023-03-16 15:35:40] conanfile.py: Generators folder: /home/lo1ol/Documents/work/rtengine
[2023-03-16 15:35:40] Install finished succesfully

I think sometimes that this can be resolved without server side modification:

If non visible require (or its dependency) has package_id_mode, which doesn't include package_id and recipe_revision, than conan may skip searching of binary package for this kind of deps.

Conan doesn't need to look for compatible package, if it doesn't affect to hash and will be skipped

lo1ol commented 1 year ago

That is a different issue

I can create another issue for this feature request:)

lo1ol commented 1 year ago

Is there some cases, when binary package is skipped and doesn't affect to hash, but still needed for search?

lo1ol commented 1 year ago

I tried to create patch for this feature and find out that adding of this feature require large refactor of code. So, you can close the issue.

Have I create another issue for server side modification request?

lo1ol commented 1 year ago

I discover custom workaround! I can add special option hide_private for each my packages. This option is True by default. But when I run conan install, I have to set this option to False. Conanfile will look like that:

from conan import ConanFile

class RtengineConanfile(ConanFile):
    settings = "os", "compiler", "arch", "build_type"

    python_requires = "rutokenconanfile/0.33@rutoken/testing"
    python_requires_extend = "rutokenconanfile.RutokenConanFile"

    def requirements(self):
        self.mytest_requires(...)
from conan import ConanFile

class RutokenConanFile:
    def init(self):
        self.options.update({"hide_private": [True, False]}, {"hide_private": True})

    def mytest_requires(self, *args, **vargs):
        if  not self.options.get_safe("hide_private"):
            self.test_requires(*args, **vargs)

class RutokenConanFile_(ConanFile):
    name = "rutokenconanfile"
    version = "0.33"
    package_type = "python-require"