[question] Install using incorrect package ID

[radonish]

I'm trying to debug what appears to be an erroneous package ID getting substituted for a valid package ID.

Let's say we have:

• libA
• appA
• appA depends on libA

I do the following:

1. Clear Conan cache folder
2. conan create libA
3. conan create appA
4. conan upload libA
5. conan upload appA
6. Clear Conan cache folder
7. conan install appA

The uploads are to a Conan repo on my company's Artifactory server. I verify the package ID that gets printed in the create operations matches:

1. Package ID printed in upload operations
2. Package ID displayed in Artifactory

Further, I confirm that appA's "Conan Package Info" view within the Artifactory UI - where it's dependencies and their package IDs are displayed - also displays a matching package ID for libA. All package IDs are matching up.

When I do the 'conan install appA', however, it displays a package ID for libA that I can find no reference to anywhere in the Artifactory repository. I don't even see an old version of the package it could be getting it from.

In reality appA in my example has many other dependencies, all of which are properly matching the expected package ID from the create and upload operations.

I'm trying to debug where this mysterious package ID is coming from for this one libA package - turning verbose logging on did not shed any light. Any help in the matter would be greatly appreciated.

Thanks

[radonish]

Please note the following package ID mode-related settings:

default_package_id_mode = package_revision_mode revisions_enabled = 1

[danimtb]

Hi @radonish,

This is really weird and should not happen unless you are doing something not usual in the recipe methods. Could you please provide me some recipe examples so I can try to reproduce it from my side?

Thanks

[radonish]

Hello @danimtb,

The recipe that seems to be the problem is based off of this Poco recipe; I essentially had to modify it to simply add a source code patch:

https://github.com/pocoproject/conan-poco/blob/master/conanfile.py

My library has Poco as a dependency. It builds successfully and references (in stdout and in Artifactory) the Poco package ID that also previously built successfully.

I've started to poke through the Conan code to try and understand where it's grabbing the bogus package ID it's claiming is the right one that my library depends on. It looks like it's within the graphing code..

Thank you

[radonish]

@danimtb, I started a build from scratch (clearing my local Conan cache and clearing the Artifactory repository) after changing revisions_enabled to 0 as it seemed that it was not necessary for package_revision_mode according to the documentation (please confirm).

Interestingly enough, I actually ran into the same flavor of problem but this time it was when attempting to build Poco itself. In this case, it was complaining about OpenSSL not being present even though I had just built it with the same profile. Further, it seems like this automated build of Poco by the maintainer captures the same issue:

https://ci.appveyor.com/project/obiltschnig/conan-poco/build/job/n2q847n1yckachvb

See line 1456.It complains it cannot find OpenSSL even though it seems to have found a package earlier in the log.

Thanks

[radonish]

@danimtb - Sorry for the spam but I think I'm getting closer; I let OpenSSL get built again by adding "--build OpenSSL" with the intention of diff'ing the conaninfo.txt files for the OpenSSL I initially built and the one that the Poco package create would now build. Here is what I found:

diff /scratch/conan_data/OpenSSL/1.0.2o/conan/stable/package/244dca2fb4877b55804f205d566a5400133b9ac9/conaninfo.txt /scratch/conan_data/OpenSSL/1.0.2o/conan/stable/package/266fabe4e483a7eddd9f053b33f3f1fb45b08cf0/conaninfo.txt 
9c9
<     zlib/1.2.11@conan/stable#0:3a5f72c8cd50641b8efa6ed13e6914c6ced2747c#607012759027ba2e6807621d4ef5a26e
---
>     zlib/1.2.11@conan/stable#0:3a5f72c8cd50641b8efa6ed13e6914c6ced2747c#0

As you can see, it seems like the issue is the zlib OpenSSL dependency having different revision values: 0 when I create the OpenSSL package via "conan install OpenSSL/1.0.2o@conan/stable --build OpenSSL" vs. when I create the OpenSSL package again via "conan create /path/to/my/custom/Poco/recipe/ Poco/1.9.0@me/stable --build OpenSSL

Since both the OpenSSL and zlib recipes are being used unmodified I'm not sure where I am going wrong.

[danimtb]

Hi @radonish,

I am trying the steps you describe but I am unable to reproduce the issue with the missing package for openssl. I will need more information:

• What profile are you using? If it is the default one, please paste the contents you get executing conan profile show default command

• How does the conanfile.txt or .py look like? I would need the exact version or the direct requirements you are using (I think it is only Poco, but I don't know the version)

---

Please make sure you have revisions_enabled = 1 and default_package_id_mode = package_revision_mode

[radonish]

@danimtb, thanks for the help.

Here is the profile I am using:

$ conan profile show profile-x86_64-rhel
Configuration for profile profile-x86_64-rhel:

[settings]
os=Linux
os_build=Linux
arch=x86_64
arch_build=x86_64
compiler=gcc
compiler.version=4.8
compiler.libcxx=libstdc++
build_type=Release
[options]
[build_requires]
[env]
CONAN_CMAKE_SYSTEM_NAME=Linux
CONAN_CMAKE_SYSTEM_PROCESSOR=x86_64
CC=gcc
CXX=g++

conanfile.py (used original Poco community file but added patches to pull in the Poco DNSSD optional library):

import os
import shutil

from conans import CMake
from conans import ConanFile
from conans import tools

class PocoConan(ConanFile):
    name = "Poco-CUSTOM"
    version = "1.9.0"
    url = "http://github.com/pocoproject/conan-poco"
    exports_sources = "CMakeLists.txt", "PocoMacros.cmake", "0001-poco-1.9.0-OpenSSL.patch", "0002-poco-1.9.0-CppUnit.patch", "0003-poco-1.9.0-DNSSD-CMake.patch", "0004-poco-1.9.0-DNSSD-Files.patch", "0005-poco-1.9.0-SignalHandler.patch", "0006-poco-1.9.0-CppUnit-AssertFix.patch"
    generators = "cmake", "txt"
    settings = "os", "arch", "compiler", "build_type"
    license = "The Boost Software License 1.0"
    description = "Modern, powerful open source C++ class libraries for building network- and internet-based " \
                  "applications that run on desktop, server, mobile and embedded systems."
    options = {"shared": [True, False],
               "fPIC": [True, False],
               "enable_xml": [True, False],
               "enable_json": [True, False],
               "enable_mongodb": [True, False],
               "enable_pdf": [True, False],
               "enable_util": [True, False],
               "enable_net": [True, False],
               "enable_netssl": [True, False],
               "enable_netssl_win": [True, False],
               "enable_crypto": [True, False],
               "enable_data": [True, False],
               "enable_data_sqlite": [True, False],
               "enable_data_mysql": [True, False],
               "enable_data_odbc": [True, False],
               "enable_sevenzip": [True, False],
               "enable_zip": [True, False],
               "enable_apacheconnector": [True, False],
               "enable_cppparser": [True, False],
               "enable_pocodoc": [True, False],
               "enable_pagecompiler": [True, False],
               "enable_pagecompiler_file2page": [True, False],
               "force_openssl": [True, False],  # "Force usage of OpenSSL even under windows"
               "enable_tests": [True, False],
               "poco_unbundled": [True, False],
               "cxx_14": [True, False],
               "enable_dnssd": [True, False],
               "enable_dnssd_avahi": [True, False]
              }
    default_options = '''
shared=True
fPIC=True
enable_xml=True
enable_json=True
enable_mongodb=True
enable_pdf=False
enable_util=True
enable_net=True
enable_netssl=True
enable_netssl_win=False
enable_crypto=True
enable_data=True
enable_data_sqlite=True
enable_data_mysql=False
enable_data_odbc=False
enable_sevenzip=False
enable_zip=True
enable_apacheconnector=False
enable_cppparser=False
enable_pocodoc=False
enable_pagecompiler=False
enable_pagecompiler_file2page=False
force_openssl=True
enable_tests=False
poco_unbundled=False
cxx_14=False
enable_dnssd=True
enable_dnssd_avahi=True
'''

    def source(self):
        zip_name = "poco-%s-release.zip" % self.version
        tools.download("https://github.com/pocoproject/poco/archive/%s" % zip_name, zip_name)
        tools.unzip(zip_name)
        shutil.move("poco-poco-%s-release" % self.version, "poco")
        os.unlink(zip_name)

        # Handle DNSSD - patch before further CMake manipulation below
        self.run("git clone --depth 1 https://github.com/pocoproject/poco-dnssd.git poco/DNSSD")
        tools.patch(patch_file="0001-poco-1.9.0-OpenSSL.patch", base_path="poco")
        tools.patch(patch_file="0002-poco-1.9.0-CppUnit.patch", base_path="poco")
        tools.patch(patch_file="0003-poco-1.9.0-DNSSD-CMake.patch", base_path="poco")
        tools.patch(patch_file="0004-poco-1.9.0-DNSSD-Files.patch", base_path="poco")
        tools.patch(patch_file="0005-poco-1.9.0-SignalHandler.patch", base_path="poco")
        tools.patch(patch_file="0006-poco-1.9.0-CppUnit-AssertFix.patch", base_path="poco")

        shutil.move("poco/CMakeLists.txt", "poco/CMakeListsOriginal.cmake")
        shutil.move("CMakeLists.txt", "poco/CMakeLists.txt")
        # Patch the PocoMacros.cmake to fix the detection of the win10 sdk.
        # NOTE: ALREADY FIXED IN POCO REPO, REMOVE THIS FOR NEXT VERSION
        shutil.move("PocoMacros.cmake", "poco/cmake/PocoMacros.cmake")

    def config_options(self):
        if self.settings.os == "Windows":
            del self.options.fPIC

    def configure(self):
        if self.options.enable_apacheconnector:
            raise Exception("Apache connector not supported: https://github.com/pocoproject/poco/issues/1764")

    def requirements(self):
        if self.options.enable_netssl or self.options.enable_netssl_win or self.options.enable_crypto or self.options.force_openssl:
            self.requires.add("OpenSSL/1.0.2o@conan/stable", private=False)

        if self.options.enable_data_mysql:
            # self.requires.add("MySQLClient/6.1.6@hklabbers/stable")
            raise Exception("MySQL not supported yet, open an issue here please: %s" % self.url)

        if self.options.enable_dnssd_avahi:
            self.requires.add("avahi/0.7@custom/stable", private=False)

    def build(self):
        if self.settings.compiler == "Visual Studio" and self.options.shared:
            self.output.warn("Adding ws2_32 dependency...")
            replace = 'Net Util Foundation Crypt32.lib'
            tools.replace_in_file("poco/NetSSL_Win/CMakeLists.txt", replace, replace + " ws2_32 ")

            replace = 'Foundation ${OPENSSL_LIBRARIES}'
            tools.replace_in_file("poco/Crypto/CMakeLists.txt", replace, replace + " ws2_32 Crypt32.lib")

        cmake = CMake(self, parallel=None)  # Parallel crashes building
        for option_name in self.options.values.fields:
            activated = getattr(self.options, option_name)
            if option_name == "shared":
                cmake.definitions["POCO_STATIC"] = "OFF" if activated else "ON"
            elif not option_name == "fPIC":
                cmake.definitions[option_name.upper()] = "ON" if activated else "OFF"

        if self.settings.os == "Windows" and self.settings.compiler == "Visual Studio":  # MT or MTd
            cmake.definitions["POCO_MT"] = "ON" if "MT" in str(self.settings.compiler.runtime) else "OFF"
        os.mkdir("build")

        cmake.verbose = False
        cmake.configure(source_dir="../poco", build_dir="build")
        self.output.info(cmake.definitions)
        cmake.build()

    def package(self):
        # Copy the license files
        self.copy("poco/LICENSE", dst=".", keep_path=False)
        # Typically includes we want to keep_path=True (default)
        packages = ["CppUnit", "Crypto", "Data", "Data/MySQL", "Data/ODBC", "Data/SQLite",
                    "DNSSD", "DNSSD/Avahi", "Foundation", "JSON", "MongoDB", "Net", "Util",
                    "XML", "Zip"]
        if self.settings.os == "Windows":
            packages.append("NetSSL_Win")
        else:
            packages.append("NetSSL_OpenSSL")

        for header in packages:
            self.copy(pattern="*.h", dst="include", src="poco/%s/include" % header)

        # But for libs and dlls, we want to avoid intermediate folders
        self.copy(pattern="*.lib", dst="lib", src="build/lib", keep_path=False)
        self.copy(pattern="*.a",   dst="lib", src="build/lib", keep_path=False)
        self.copy(pattern="*.dll", dst="bin", src="build/bin", keep_path=False)
        # in linux shared libs are in lib, not bin
        self.copy(pattern="*.so*", dst="lib", src="build/lib", keep_path=False, symlinks=True)
        self.copy(pattern="*.dylib", dst="lib", src="build/lib", keep_path=False)

    def package_info(self):
        """ Define the required info that the consumers/users of this package will have
        to add to their projects
        """
        libs = [("enable_mongodb", "PocoMongoDB"),
                ("enable_pdf", "PocoPDF"),
                ("enable_net", "PocoNet"),
                ("enable_netssl", "PocoNetSSL"),
                ("enable_netssl_win", "PocoNetSSLWin"),
                ("enable_crypto", "PocoCrypto"),
                ("enable_data", "PocoData"),
                ("enable_data_sqlite", "PocoDataSQLite"),
                ("enable_data_mysql", "PocoDataMySQL"),
                ("enable_data_odbc", "PocoDataODBC"),
                ("enable_sevenzip", "PocoSevenZip"),
                ("enable_zip", "PocoZip"),
                ("enable_apacheconnector", "PocoApacheConnector"),
                ("enable_util", "PocoUtil"),
                ("enable_xml", "PocoXML"),
                ("enable_json", "PocoJSON"),
                ("enable_dnssd", "PocoDNSSD"),
                ("enable_dnssd_avahi", "PocoDNSSDAvahi")]

        suffix = str(self.settings.compiler.runtime).lower()  \
                 if self.settings.compiler == "Visual Studio" and not self.options.shared \
                 else ("d" if self.settings.build_type=="Debug" else "")
        for flag, lib in libs:
            if getattr(self.options, flag):
                if self.settings.os == "Windows" and flag == "enable_netssl":
                    continue
                if self.settings.os != "Windows" and flag == "enable_netssl_win":
                    continue
                self.cpp_info.libs.append("%s%s" % (lib, suffix))

        self.cpp_info.libs.append("PocoFoundation%s" % suffix)
        self.cpp_info.libs.append("CppUnit%s" % suffix)

        # in linux we need to link also with these libs
        if self.settings.os == "Linux":
            self.cpp_info.libs.extend(["pthread", "dl", "rt"])

        if not self.options.shared:
            self.cpp_info.defines.extend(["POCO_STATIC=ON", "POCO_NO_AUTOMATIC_LIBS"])
            if self.settings.compiler == "Visual Studio":
                self.cpp_info.libs.extend(["ws2_32", "Iphlpapi.lib", "Crypt32.lib"])

[danimtb]

Ok, I think that I understand now what is happening.

When using default_package_id_mode = package_revision_mode you are using the most restrictive way of dependency relationship. This means that the requirements are binary specific.

In your case the dependency graph is: Poco --> OpenSSL --> zlib

When you try to create a package for Poco, Conan needs to give to the Poco package you are creating a package ID. To do this, Conan needs the OpenSSL package ID (missing). However, the OpenSSL package ID is missing because it needs the zlib's package ID (you already have it).

So, what is the issue then?

There is no OpenSSL package available with package_revision_mode requirement of zlib. This is the reason you have to do --build OpenSSL.

Why it does not work out-of-the-box?

Conan uses semver_direct_mode by default to calculate the package IDs. This means that there is not such a tight relationship in the requirements as package revisions and recipe revisions are not taken into account. We cannot make this package id mode the default because any update in requirements such as zlib will require a full rebuild of all the packages of all libraries in Conan Center.

If you want more information about this please read carefully the blog post you pointed above: https://blog.conan.io/2019/09/27/package-id-modes.html and beware that there are important implications regarding the package ID you choose.

Hope it makes sense. Thanks 😄

[radonish]

@danimtb, thank you for the details. My steps are as follows:

1. conan install zlib/1.2.11@conan/stable --build zlib -pr=profile-x86_64-rhel
2. conan install OpenSSL/1.0.2o@conan/stable --build OpenSSL -pr=profile-x86_64-rhel
3. conan create /path/to/my/custom/poco/ Poco-CUSTOM/1.9.0@custom/stable -pr=profile-x86_64-rhel

Wouldn't steps 1 and 2 ensure that my OpenSSL and zlib packages were built with package_revision_mode since I forced a fresh build with my profile?

Thanks

[radonish]

@danimtb, I changed the procedure above:

1. conan install zlib/1.2.11@ --build zlib -pr=profile-x86_64-rhel
2. conan install openssl/1.0.2t@ --build openssl -pr profile-x86_64-rhel
3. (unchanged)

It appears that "OpenSSL/1.0.2o" vs. "openssl/1.0.2t" was the key to making things work in package_revision_mode. The other possibility is that leaving out the user/channel in the reference was key - I left it out as it was the only way to get the build to succeed with the updated openssl recipe.

I'm kicking off a full build from scratch to have a full confirmation; I will close the issue if it succeeds.

I appreciate your support.

[radonish]

@danimtb, my full build hierarchy did successfully complete. The next phase of my test was to: 1) Upload all packages to my company's Artifactory Conan repository 2) Clear my local Conan cache 3) Attempt to build a package, hoping all dependencies are pulled from my company's Artifactory Conan repository (binaries)

Unfortunately, step 3 did not succeed. Once again packages that should have been found on my remote came up as unknown/missing. Doing some research, it looks like Artifactory may not have supported package revisions completely until version 6.9 - we are using 6.4.1. I am waiting for our EIT group to upgrade the server and will report back after the holidays.

[danimtb]

thanks for the detailed steps. As mentioned above, this is related to the package_revision_mode and the way the package ids are calculated

[radonish]

@danimtb, I have confirmed that my primary issue was related to the version of Artifactory being used. I updated to the latest version, that has full support for package revisions, and everything behaved as it should.

Closing the issue. Thanks for the support!

[danimtb]

@radonish glad you finally figured out what was happening! Please do not hesitate to open an issue if you have more questions or something is not working. We will be happy to help 😄