[bug] Build of a recipe which has a tool_requires with `<host_version>` may fail if build triggered from build context

SpaceIm commented 1 year ago

Environment details

Operating System+version: Ubuntu 22.04
Compiler+version: GCC 12
Conan version: 2.0.13
Python version: 3.10.5

Steps to reproduce

For example you could add qt recipe to build requirements of a recipe on Linux, it will drag xkbcommon in build context. Also take care of not downloading pre-built binaries from conancenter but build from source, and do not have xkbcommon in host context (or a different flavor like shared=True). Build of xkbcommon will fail (It was working fine before https://github.com/conan-io/conan-center-index/pull/19983 which has replaced wayland/1.21.0 by wayland/<host_version> in its build requirements).

If you build xkbcommon with a simple conan install --requires xkbcommon/1.5.0 -b, it works fine.

It might be an issue with PkgConfigDeps build_context_activated with <host_version in the context of a build triggered from build context, I don't know: https://github.com/conan-io/conan-center-index/blob/8d3eb5dcf5fcf29d971892e156781b5e26e1453a/recipes/xkbcommon/all/conanfile.py#L104-L138

Logs

xkbcommon/1.5.0: Calling build()
xkbcommon/1.5.0: Meson configure cmd: meson setup --native-file "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release/conan/conan_meson_native.ini" "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release" "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/src" -Dprefix="/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/p"
xkbcommon/1.5.0: RUN: meson setup --native-file "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release/conan/conan_meson_native.ini" "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release" "/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/src" -Dprefix="/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/p"
The Meson build system
Version: 1.2.2
Source dir: /home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/src
Build dir: /home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release
Build type: native build
Project name: libxkbcommon
Project version: 1.5.0
C compiler for the host machine: /usr/bin/gcc-12 (gcc 12.3.0 "gcc-12 (Ubuntu 12.3.0-1ubuntu1~22.04) 12.3.0")
C linker for the host machine: /usr/bin/gcc-12 ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -fno-strict-aliasing: YES
Compiler for C supports arguments -Wno-unused-parameter: YES
Compiler for C supports arguments -Wno-missing-field-initializers: YES
Compiler for C supports arguments -Wpointer-arith: YES
Compiler for C supports arguments -Wmissing-declarations: YES
Compiler for C supports arguments -Wformat=2: YES
Compiler for C supports arguments -Wstrict-prototypes: YES
Compiler for C supports arguments -Wmissing-prototypes: YES
Compiler for C supports arguments -Wnested-externs: YES
Compiler for C supports arguments -Wbad-function-cast: YES
Compiler for C supports arguments -Wshadow: YES
Compiler for C supports arguments -Wlogical-op: YES
Compiler for C supports arguments -Wdate-time: YES
Compiler for C supports arguments -Wwrite-strings: YES
Compiler for C supports arguments -Wno-documentation-deprecated-sync: NO
Found pkg-config: /home/spaceim/.conan2/p/b/pkgco720a404f4464b/p/bin/pkgconf (2.0.3)
Run-time dependency xkeyboard-config found: YES 2.33
Has header "unistd.h" : YES
Checking if "__builtin_expect" : links: YES
Header "unistd.h" has symbol "eaccess" : YES
Header "unistd.h" has symbol "euidaccess" : YES
Header "sys/mman.h" has symbol "mmap" : YES
Header "stdlib.h" has symbol "mkostemp" : YES
Header "fcntl.h" has symbol "posix_fallocate" : YES
Header "string.h" has symbol "strndup" : YES
Header "stdio.h" has symbol "asprintf" : YES
Header "stdlib.h" has symbol "secure_getenv" : YES
Header "limits.h" has symbol "PATH_MAX" : YES
Checking if "-Wl,--version-script" : links: YES
Program scripts/map-to-def found: YES (/home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/src/scripts/map-to-def)
Program bison found: YES (/home/spaceim/.conan2/p/b/bison7391da5f74f3d/p/bin/bison)
Run-time dependency xcb found: YES 1.14
Run-time dependency xcb-xkb found: YES 1.14
Run-time dependency libxml-2.0 found: YES 2.11.4
Header "getopt.h" has symbol "getopt_long" : YES
Has header "linux/input.h" : YES
Found CMake: /usr/bin/cmake (3.27.6)
Run-time dependency wayland-client found: NO (tried pkgconfig and cmake)
Found pkg-config: /home/spaceim/.conan2/p/b/pkgco720a404f4464b/p/bin/pkgconf (2.0.3)
Build-time dependency wayland-protocols_build found: YES 1.31
Build-time dependency wayland-scanner_build found: YES 1.22.0

../src/meson.build:481:12: ERROR: Problem encountered: The Wayland xkbcli programs require wayland-client >= 1.2.0, wayland-protocols >= 1.7 which were not found.
You can disable the Wayland xkbcli programs with -Denable-wayland=false.

A full log can be found at /home/spaceim/.conan2/p/b/xkbco9f60d54155a8e/b/build-release/meson-logs/meson-log.txt

xkbcommon/1.5.0: ERROR:
Package 'f9ef332b3b3d0aa590e813e2022896740eb3a979' build failed

memsharded commented 1 year ago

Thanks for reporting.

If the <host_version> is being resolved to a real version and it matches the correct one, then this doesn't seem related? So maybe this fail is because of that specific package being built as a dependency, not because of the <host_version> things.

I cannot reproduce this at the moment, this package doesn't work in Windows, and I don't have linux right now. It would be great to have some minimal reproducible case.

SpaceIm commented 1 year ago

Is there not some hidden bug? Sure it resolves to the correct version, but the build fails because it seems to be slightly different than the same build triggered from host context (this one succeeds), and it shouldn't because I have exactly the same profile in host & build context. This is why I suspect a bug when a build is triggered from build context. And because the same build was working fine before this <host_version>, yes I suspect something related.

It's hard to provide a reproducible example except using current RREV of xkbcommon in conancenter. The best I can do is to provide a diff of all files generated in conan folder of build folder, between a regular build of xbkcommon with conan install --requires xkbcommon, and an "indirect build" of xkbcommon from build context (with host & build profiles identical). If the diff is not null, there is a bug. It wouldn't be the first time (I don't find previous issue, but I remember an old bug of builds from build context).

SpaceIm commented 1 year ago

If you are on Windows, you can build xkbcommon in WSL.

Step 1: Check that it builds fine with a regular installation:

conan install --requires xkbcommon/1.5.0 -b xkbcommon -b missing

Step 2: check that it fails if build is triggered from build context. These commands may be sufficient:

conan remove xkbcommon/1.5.0 -c
conan install --tool-requires xkbcommon/1.5.0 -b xkbcommon -b missing

Since host & build profile are the same, the second build should be exactly the same, but it fails.

memsharded commented 1 year ago

It seems to work fine in Conan 2.0.13 in WSL:

conan remove "*" -c
conan install --tool-requires xkbcommon/1.5.0 -b xkbcommon
....
cli: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
Install finished successfully

jwillikers commented 1 year ago

@memsharded I've also ran into this exact issue using Conan V1.

../src/meson.build:481:12: ERROR: Problem encountered: The Wayland xkbcli programs require wayland-client >= 1.2.0, wayland-protocols >= 1.7 which were not found.
You can disable the Wayland xkbcli programs with -Denable-wayland=false.

SpaceIm commented 1 year ago

Yep conan install --tool-requires xkbcommon/1.5.0 -b xkbcommon -b missing may not be enough to reproduce this issue. Try to add xkbcommon to dependencies of a recipe which is a build requirement of another recipe.

ericLemanissier commented 1 year ago

It seems more probable that this is a bug in xkbcommon than in conan itself, considering the recipes does some different actions at 5 different steps, depending on if there is a build profile or not : https://github.com/search?q=repo%3Aconan-io%2Fconan-center-index+_has_build_profile+path%3A%2F%5Erecipes%5C%2Fxkbcommon%5C%2Fall%5C%2F%2F&type=code

jwillikers commented 1 year ago

It seems more probable that this is a bug in xkbcommon than in conan itself, considering the recipes does some different actions at 5 different steps, depending on if there is a build profile or not : https://github.com/search?q=repo%3Aconan-io%2Fconan-center-index+_has_build_profile+path%3A%2F%5Erecipes%5C%2Fxkbcommon%5C%2Fall%5C%2F%2F&type=code

I've definitely had the same thought, and finally opened issue #14935 to try to simplify this kind of thing, which I should have done a long time ago.

SpaceIm commented 11 months ago

Still broken in conan 2.0.15

It's not a recipe issue, otherwise why it doesn't fail with conan install --requires xkbcommon/1.6.0, but fails if you build let's say a recipe depending on qt in build requirements (dragging indirectly xkbcommon in build context), like qwt with all shared in host context, but static in build context?

memsharded commented 11 months ago

A couple of summary points here:

This seems more related to PkgConfigDeps than to the <host_version>, but this needs to be checked
It would still be necessary a minimal way to reproduce it, preferably without bringing qt or other huge packages there, to reduce the area. If the issue comes because something is a tool-requires of something, then a 5 lines recipe with that requirement should help to reproduce it.

jwillikers commented 9 months ago

I think you can create a minimal example by creating a simple Conan package for a command-line tool that depends on xkbcommon. Then, add a test a package to run the tool as a tool_requires and you should get the error. I hit this issue in conan-io/conan-center-index#22412.

memsharded commented 9 months ago

Thanks for the feedback @jwillikers

I have just tried exactly what you propose with latest Conan 2.0.17 and it works fine, I cannot make it fail.

This is why I say that we need something minimal (that is still far from minimal, for a start it only works in Linux, so I cannot test things in my default system which is Windows) to reproduce, like exact steps and commands, even including docker images (I tried using the conanio/gcc11 docker image)

jwillikers commented 9 months ago

I forgot to mention that you have to have the option xkbcommon/*:with_wayland=True.

memsharded commented 9 months ago

I forgot to mention that you have to have the option xkbcommon/*:with_wayland=True

Still the same, it seems to work.

jwillikers commented 9 months ago

I forgot to mention that you have to have the option xkbcommon/*:with_wayland=True

Still the same, it seems to work.

@memsharded Thanks for looking into this. I'll work on getting a minimal example of the issue when I get time, though I haven't checked if the problem occurs with Conan v2 yet.

blockspacer commented 8 months ago

Thanks for reporting.

If the <host_version> is being resolved to a real version and it matches the correct one, then this doesn't seem related? So maybe this fail is because of that specific package being built as a dependency, not because of the <host_version> things.

I cannot reproduce this at the moment, this package doesn't work in Windows, and I don't have linux right now. It would be great to have some minimal reproducible case.

@memsharded

Got same wayland errors during Conan v2 install with self.requires("qt/6.6.1"), -s build_type=Debug and -o *:shared=True in conanfile (Ubuntu):

conan install . --profile:build=changeme_profile --profile:host=changeme_profile -c tools.system.package_manager:mode=install -c tools.system.package_manager:sudo=True --build=missing -o :shared=True -o abseil:shared=False -o boost:shared=True -o qt:shared=True -c tools.build:verbosity=verbose -c tools.build:jobs=4 -s compiler.cppstd=gnu17 -c tools.build:skip_test=True -s build_type=Debug

NOTE: I had to also add "qt/6.6.1" to tool_requires due to $env:PATH without QT paths after .\build\generators\conanbuild.ps1 i.e. without moc.exe on windows (probably separate bug) self.tool_requires("qt/6.6.1") # moc.exe requires to set$env:QT_PLUGIN_PATH,$env:QT_HOST_PATH, `$env:PATH```

Error logs and conanfile (ignore comments in conanfile) below:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y python3 python3-pip python3-dev python-dev-is-python3
sudo apt-get install -y curl lsb-release libncurses5 binutils autoconf pkg-config autotools-dev git gcc make nano cmake apt-transport-https ca-certificates wget
sudo apt-get install -y mesa-utils libglu1-mesa-dev freeglut3-dev mesa-common-dev
sudo apt-get install -y libx11-dev libx11-xcb-dev libfontenc-dev libice-dev libsm-dev libxau-dev libxaw7-dev
sudo apt-get install -y libxcomposite-dev libxcursor-dev libxdamage-dev libxfixes-dev libxi-dev libxinerama-dev libxkbfile-dev libxmuu-dev libxrandr-dev libxrender-dev libxres-dev libxss-dev libxtst-dev libxv-dev libxxf86vm-dev libxcb-render0-dev libxcb-render-util0-dev libxcb-xkb-dev libxcb-icccm4-dev libxcb-image0-dev libxcb-keysyms1-dev libxcb-randr0-dev libxcb-shape0-dev libxcb-sync-dev libxcb-xfixes0-dev libxcb-xinerama0-dev libxcb-dri3-dev uuid-dev libxcb-cursor-dev
sudo apt-get install -y libxcb-util-dev

Python version 3.6 or later is required by boost and qt

python3 --version

python3 -m pip install --upgrade pip

# https://github.com/conan-io/conan-center-index/issues/11489
pip3 install html5lib

# use latest Conan 2 version
pip3 install conan --upgrade

conan profile detect

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux

from conan import ConanFile, tools
from conan.errors import ConanInvalidConfiguration, ConanException
import os, re, stat, fnmatch, platform, glob, traceback, shutil
from functools import total_ordering

import sys
import os
import shutil
from io import StringIO

from conan import ConanFile, tools
from conan.errors import ConanException

from conan.tools.cmake import CMake, CMakeToolchain, CMakeDeps, cmake_layout
from conan.tools.env import Environment
from conan.tools.build import can_run, check_max_cppstd, check_min_cppstd
from conan.tools.env import VirtualRunEnv, VirtualBuildEnv
from conan.tools.files import copy, save, load, rmdir
from conan.tools.microsoft import is_msvc_static_runtime, is_msvc
from pathlib import Path
from conan.tools.layout import basic_layout
from conan.tools.scm import Version

# conan runs the methods in this order:
# config_options(),
# configure(),
# requirements(),
# package_id(),
# build_requirements(),
# build_id(),
# system_requirements(),
# source(),
# imports(),
# build(),
# package(),
# package_info()

required_conan_version = ">=2.1.0"

class ModuleConan(ConanFile):
  settings = "os", "compiler", "build_type", "arch"
  # "CMakeToolchain", "CMakeDeps", "VirtualRunEnv", "VirtualBuildEnv"
  #generators = "PkgConfigDeps"
  #package_type = "application"
  # https://github.com/conan-io/conan/issues/15718#issuecomment-1957064874
  #package_type = "library"
  #package_type = "static-library"
  #package_type = "shared-library"
  options = {
    "cxx_modules": [True, False],
    "shared": [True, False], 
    "fPIC": [True, False],
    "python_executable": [None, "ANY"],  # system default python installation is used, if None
    "python_version": [None, "ANY"],  # major.minor; computed automatically, if None
  }
  default_options = {
    "python_executable": None,
    "python_version": None,
    "cxx_modules": False,
    "shared": True, 
    "fPIC": True,
    "qt*:qt5compat": False,
    "qt*:with_vulkan": False,
    "qt*:with_icu": True,
    #"qt*:with_glib": True,
    #"qt*:with_mysql": True,
    #"qt*:with_odbc": True,
    #"qt*:with_pq": True,
    #"qt*:with_openal": True,
    #"qt*:qtserialport": True,
    "qt*:with_sqlite3": True,
    "qt*:with_sdl2": True,
    "qt*:qttools": True,
    "qt*:qtlottie": True,
    "qt*:qtquicktimeline": True,
    "qt*:qtquick3d": True,
    "qt*:qtshadertools": True,
    "qt*:qtvirtualkeyboard": True,
    "qt*:qtwebengine": False, # requires Python3 html5lib
    "qt*:qtwayland": True,
    "qt*:qtwebview": False,
    "qt*:qthttpserver": True,
    "qt*:qtgrpc": True,
    "qt*:qtwebsockets": False,
    "qt*:qtwebchannel": False,
    "qt*:qtremoteobjects": True,
    "qt*:qtsvg": True,
    "qt*:qtquickcontrols2": True,
    "qt*:qtdeclarative": True,
    "qt*:gui": True,
    #"qt*:opengl": "desktop",
    "qt*:qtcharts": True,
    "qt*:qtgraphs": True,
    #"qt*:with_libjpeg": "libjpeg-turbo",
    "qt*:qtshadertools": True,
    "qt*:qttranslations": True,
    "qt*:qtmultimedia": True,
    "qt*:qtimageformats": True,
    #"boost*:without_fiber": False,
    #"boost*:without_mpi": False, # https://www.boost.org/doc/libs/1_72_0/doc/html/mpi/getting_started.html
    # "boost*:without_python": False, # https://www.boost.org/doc/libs/1_70_0/libs/python/doc/html/building/configuring_boost_build.html
    #"qwt*:shared": True,
    #"poco*:shared": True,
    #"qt*:shared": True,
    #"boost*:shared": True,
    #"boost*:use_icu": True,
    #"openssl*:shared": True,
    #"zlib*:shared": True,
    #"libpng*:shared": True,
    #"fmt*:shared": True,
    #"hdf5*:shared": True,
    #"gsl*:shared": True,
    "gtest*:build_mock": True,
    "gtest*:build_gmock": True,
    "sol2*:with_lua": "luajit",
    "glad*:gl_profile": "core",
    "glad*:gl_version": "3.3",
    "glad*:spec": "gl",
    "glad*:no_loader": False
  }
  exports = "LICENSE.md"
  exports_sources = (
    "!build/**", 
    "!cmake-build-*/**"
    "docs/**",
    "src/**",
    "test/**",
    "tests/**",
    "cmake/**",
    "example/**",
    "examples/**",
    "benchmarks/**", 
    "benchmark/**", 
    "tools/**",
    "tool/**",
    "share/**", 
    "third_party/**",
    "VERSION", 
    "CMakeLists.txt", 
    "config.h.in",
  )

  @property
  def _min_cppstd(self):
    return "17"

  @property
  def _compilers_minimum_version(self):
    return {
      "14": {
        "Visual Studio": "14",
        "msvc": "190",
        "gcc": "5",
        "clang": "3.2",
        "apple-clang": "4.3",
      },
      "17": {
        "Visual Studio": "15" if Version(self.version) < "3.3.0" else "16",
        "msvc": "191" if Version(self.version) < "3.3.0" else "192",
        "gcc": "7",
        "clang": "6",
        "apple-clang": "10",
      },
    }.get(self._min_cppstd, {})

  def validate(self):
    if self.settings.compiler.get_safe("cppstd"):
      check_min_cppstd(self, self._min_cppstd)
    def loose_lt_semver(v1, v2):
      lv1 = [int(v) for v in v1.split(".")]
      lv2 = [int(v) for v in v2.split(".")]
      min_length = min(len(lv1), len(lv2))
      return lv1[:min_length] < lv2[:min_length]
    minimum_version = self._compilers_minimum_version.get(str(self.settings.compiler), False)
    if minimum_version and loose_lt_semver(str(self.settings.compiler.version), minimum_version):
      raise ConanInvalidConfiguration(
          f"{self.ref} requires C++{self._min_cppstd}, which your compiler does not support."
      )
    check_max_cppstd(self, "22")
    if Version(self._python_version) < self._min_python_version:
      raise ConanInvalidConfiguration(
          f"{self.ref} requires python{self._min_python_version}."
      )

  #def _version(self):
  #  if not self.version:
  #    buf = StringIO()
  #    self.run("git describe --always --dirty", output=buf)
  #    self.version = buf.getvalue().strip()
  #    if self.version[0] == 'v':
  #      self.version = self.version[1:]
  #  return self.version

  def set_version(self):
    self.version = load(self, Path(self.recipe_folder) / "VERSION").strip()

  def layout(self):
    #self.folders.generators = os.path.join("build")
    #self.folders.build = "build"
    #self.folders.generators = "conan"
    #basic_layout(self, src_folder="src")
    #cmake_layout(self, src_folder="src")
    cmake_layout(self)

  def generate(self):
    #path = self.dependencies["qt"].package_folder.replace("\\", "/")
    #folder = os.path.join(path, "bin")
    #bin_folder = "bin" if self.settings.os == "Windows" else "libexec"
    #save(self, "qt.conf", f"""[Paths]
#Prefix = {path}
#ArchData = {folder}/archdatadir
#HostData = {folder}/archdatadir
#Data = {folder}/datadir
#Sysconf = {folder}/sysconfdir
#LibraryExecutables = {folder}/archdatadir/{bin_folder}
#HostLibraryExecutables = bin
#Plugins = {folder}/archdatadir/plugins
#Imports = {folder}/archdatadir/imports
#Qml2Imports = {folder}/archdatadir/qml
#Translations = {folder}/datadir/translations
#Documentation = {folder}/datadir/doc
#Examples = {folder}/datadir/examples""")
    ms = VirtualBuildEnv(self)
    ms.generate()
    deps = CMakeDeps(self)
    #deps.set_property("wayland", "cmake_file_name", "Wayland")
    deps.check_components_exist = True
    deps.generate()
    vbe = VirtualBuildEnv(self)
    vbe.generate()
    vre = VirtualRunEnv(self)
    #vre.generate(scope="build")
    vre.generate()
    if can_run(self):
      VirtualRunEnv(self).generate(scope="build")
    #tc = CMakeToolchain(self)
    tc = CMakeToolchain(self, generator="Ninja")
    env = Environment()
    #env.define("MY_ENV_VAR", "MY_ENV_VAR_VALUE")
    #env = env.vars(self, scope="run")
    #env.save_script("other_env")
    tc.presets_run_environment = vre.environment().compose_env(env)
    tc.presets_build_environment = vbe.environment()
    #tc.user_presets_path = 'ConanPresets.json'
    tc.variables["CONAN_PACKAGE_VERSION"] = self.version
    #if self._build_all:
    #  tc.cache_variables["CMAKE_VERIFY_INTERFACE_HEADER_SETS"] = True
    if self.options.cxx_modules:
      tc.cache_variables["CMAKE_CXX_SCAN_FOR_MODULES"] = True
    #self._set_cmake_defs(tc.variables)
    if is_msvc(self):
      tc.variables["USE_MSVC_RUNTIME_LIBRARY_DLL"] = not is_msvc_static_runtime(self)
    tc.generate()

  def build(self):
    #self._build_with_qmake()
    self._build_with_cmake_find_package_multi()

  @property
  def _python_executable(self):
    """
    obtain full path to the python interpreter executable
    :return: path to the python interpreter executable, either set by option, or system default
    """
    exe = self.options.python_executable if self.options.python_executable else sys.executable
    return str(exe).replace('\\', '/')

  def _run_python_script(self, script):
    """
    execute python one-liner script and return its output
    :param script: string containing python script to be executed
    :return: output of the python script execution, or None, if script has failed
    """
    output = StringIO()
    command = f'"{self._python_executable}" -c "{script}"'
    self.output.info(f"running {command}")
    try:
        self.run(command, output, scope="run")
    except ConanException:
        self.output.info("(failed)")
        return None
    output = output.getvalue()
    # Conan is broken when run_to_output = True
    if "\n-----------------\n" in output:
        output = output.split("\n-----------------\n", 1)[1]
    output = output.strip()
    return output if output != "None" else None

  @property
  def _min_python_version(self):
    # Python version 3.6 or later is required by boost and qt
    return "3.6.0"

  def _detect_python_version(self):
    """
    obtain version of python interpreter
    :return: python interpreter version, in format major.minor
    """
    return self._run_python_script("from __future__ import print_function; "
                                    "import sys; "
                                    "print('{}.{}'.format(sys.version_info[0], sys.version_info[1]))")

  @property
  def _python_version(self):
    version = self._detect_python_version()
    if self.options.python_version and version != self.options.python_version:
        raise ConanInvalidConfiguration(f"detected python version {version} doesn't match conan option {self.options.python_version}")
    return version

  def requirements(self):
    # orbitprofiler
    #self.requires("pybind11/2.10.0")
    #self.requires("pyqt/6.4.0")
    #self.requires("pugixml/1.13")
    #self.requires("dcmtk/3.6.6")
    #self.requires("ftgl/2.4.0")
    #self.requires("opencascade/7.6.2")
    #self.requires("cpr/1.9.0")
    # spix "UI test automation library for QtQuick/QML Apps"
    #self.requires("glm/cci.20230113")
    #self.requires("abseil/20230802.1")
    #self.requires("opencv/4.8.1")
    self.requires("qt/6.6.1")
    #self.requires("qwt/[~6]")
    #self.requires("fmt/10.2.1")
    #self.requires("taocpp-pegtl/3.2.7")
    #self.requires("magic_enum/0.9.5")
    #self.requires("range-v3/0.12.0")
    # chromium rpc
    # chromium-v8 https://github.com/conan-io/conan-center-index/issues/6717 https://github.com/inexorgame-obsolete/conan-v8 https://github.com/luizgabriel/conan-v8
    #self.requires("strong_type/v13")
    #self.requires("gsl-lite/0.41.0")
    #self.requires("hdf5/[~1.10]")
    #self.requires("gsl/2.7.1")
    #self.requires("boost/1.84.0")
    #self.requires("zlib/1.3.1")
    #self.requires("libzip/1.10.1")
    #self.requires("sdl/2.28.5")
    #self.requires("sqlite3/3.45.0")
    #self.requires("protobuf/3.21.12")
    #self.requires("double-conversion/3.3.0")
    #self.requires("eigen/3.4.0")
    #self.requires("tesseract/5.3.3")
    #self.requires("freetype/2.13.2", override=True)
    #self.requires("harfbuzz/8.3.0")
    #self.requires("libpng/1.6.42", override=True)
    #self.requires("expat/2.5.0")
    #self.requires("libjpeg-turbo/3.0.2")
    #self.requires("libuuid/1.0.3")
    #self.requires("rapidjson/1.1.0")
    #self.requires("nlohmann_json/3.11.3")
    #self.requires("breakpad/2023.01.27") # breakpad/cci.20210521
    #self.requires("crashpad/cci.20220219")
    self.requires("catch2/3.5.2")
    self.requires("benchmark/1.8.3")
    # https://www.reddit.com/r/cpp/comments/x71tbp/suggestion_for_generalpurpose_c_libraries/
    #self.requires("folly/2022.01.31.00")
    # Poco
    #self.requires("perfetto/42.0")
    #self.requires("spdlog/1.13.0")
    #self.requires("toml11/3.8.1")
    #self.requires("imgui/1.90.4")
    #self.requires("glog/0.7.0")
    #self.requires("gflags/2.2.2")
    #self.requires("icu/74.2")
    # cling
    # jinja2cpp
    # jwasm
    # gperf/3.1
    #self.requires("tinyxml2/10.0.0")
    #self.requires("lua/5.4.6")
    #self.requires("luajit/2.1.0-beta3")
    #self.requires("yaml-cpp/0.8.0")
    #self.requires("asio/1.29.0")
    #self.requires("doctest/2.4.11")
    #self.requires("sol2/3.3.1")
    #self.requires("glad/0.1.36")
    #self.requires("glfw/3.3.8")
    #self.requires("entt/3.13.0")
    #self.requires("outcome/2.2.8")
    # libprotobuf-mutator
    #self.requires("concurrentqueue/1.0.4")
    #self.requires("glew/2.2.0")
    #self.requires("stb/cci.20230920")
    #self.requires("lz4/1.9.4")
    #self.requires("cityhash/cci.20130801")
    #self.requires("clickhouse-cpp/2.5.1")
    #self.requires("libsodium/cci.20220430")
    #self.requires("beauty/1.0.3") # HTTP Server above Boost.Beast
    #self.requires("corrade/2020.06")
    #self.requires("cpp-jwt/1.4")

    #self.requires("openssl/3.2.1")
    #self.requires("incbin/cci.20211107")

    #self.requires("qtbase/6.2.4@qt/everywhere")
    #self.requires("qtdeclarative/6.2.4@qt/everywhere")
    #if platform.system() == "Windows":
    #  self.requires("qt/6.2.4")
    #else:
    #  self.requires("qt/6.2.4")
    #  self.requires("harfbuzz/4.2.0")
    #  self.requires("openssl/1.1.1n")
    # llvm-core

    # Testing only dependencies below
    self.requires("gtest/[~1.14]")
    # g3log
    # flatbuffers
    # userver
    # http_parser
    # jemalloc
    # grpc
    # libev
    # c-ares
    # libcurl
    # cctz
    # hiredis # redis
    # amqp-cpp # rebbitmq
    # mongo-c-delf.optionsriver
    # libpq # postgresql
    #self.requires("fakeit/2.4.0")
    #self.requires("gmock/[~1.14]")

  def config_options(self):
    if self.settings.os == "Windows":
      del self.options.fPIC

  # TODO: Trying to define defaults in configure() method is no longer allowed in Conan 2.0
  def configure(self):
    #if not self.options.python_version: # if not self.options.without_python:
    #  self.options.python_version = self._detect_python_version()
    #  self.options.python_executable = self._python_executable
    if self.options.shared:
      self.options.rm_safe("fPIC")
    #self.options["boost"].python_version = self._python_version
    #self.options["opencv"].build_with_python_version = self._python_version
    #self.options["pyqt-sip"].build_with_python_version = self._python_version
    #self.options["pyqt"].build_with_python_version = self._python_version
    if self.settings.os == "Windows":
      self.options["qt"].with_glib = False
      self.options["qt"].with_harfbuzz = False
      self.options["qt"].opengl = "dynamic"

  def build_requirements(self):
    self.tool_requires("cmake/[>=3.28.1 <4]")
    self.tool_requires("ninja/1.11.1")
    self.tool_requires("qt/6.6.1") # moc.exe requires to set `$env:QT_PLUGIN_PATH`, `$env:QT_HOST_PATH, `$env:PATH``
    # build_requirements grpc_codegen
    # build_requirements protoc_installer

  #def imports(self):
  #  dest = os.getenv("CONAN_IMPORT_PATH", "bin")
  #  self.copy("crashpad_handler*", src="@bindirs",
  #            dst=dest, root_package="crashpad")
  #if not self.options.system_qt:
  #    chromium_licenses = conan_helpers.gather_chromium_licenses(self.deps_cpp_info["qt"].rootpath)
  #    chromium_licenses.sort(key=lambda license_info: license_info["name"].lower())
  #    with open(os.path.join(self.install_folder, "NOTICE.Chromium.csv"), "w") as fd:
  #        writer = csv.DictWriter(fd, fieldnames=["name", "url", "license"], extrasaction='ignore')
  #        writer.writeheader()
  #        for license in chromium_licenses:
  #            writer.writerow(license)
  #    with open(os.path.join(self.install_folder, "NOTICE.Chromium"), "w") as fd:
  #        for license in chromium_licenses:
  #            fd.write("================================================================================\n")
  #            fd.write("Name: {}\n".format(license["name"]))
  #            fd.write("URL: {}\n\n".format(license.get("url", "")))
  #            fd.write(open(license["license file"], 'r').read())
  #            fd.write("\n\n")
  #        fd.write("================================================================================\n")

  def _is_mingw(self):
      return self.settings.os == "Windows" and self.settings.compiler == "gcc"

  def _qmake_supported(self):
      return self.settings.compiler != "Visual Studio" or self.options["qt"].shared

  def _cmake_multi_supported(self):
      return True

  #def _build_with_qmake(self):
  #    if not self._qmake_supported():
  #        return
  #    tools.mkdir("qmake_folder")
  #    with tools.chdir("qmake_folder"):
  #        self.output.info("Building with qmake")
  #        with tools.vcvars(self.settings) if self.settings.compiler == "Visual Studio" else #tools.no_op():
  #            args = [self.source_folder, "DESTDIR=bin"]
  #            def _getenvpath(var):
  #                val = os.getenv(var)
  #                if val and tools.os_info.is_windows:
  #                    val = val.replace("\\", "/")
  #                    os.environ[var] = val
  #                return val
  #            value = _getenvpath('CC')
  #            if value:
  #                args.append('QMAKE_CC="%s"' % value)
  #            value = _getenvpath('CXX')
  #            if value:
  #                args.append('QMAKE_CXX="%s"' % value)
  #            value = _getenvpath('LD')
  #            if value:
  #                args.append('QMAKE_LINK_C="%s"' % value)
  #                args.append('QMAKE_LINK_C_SHLIB="%s"' % value)
  #                args.append('QMAKE_LINK="%s"' % value)
  #                args.append('QMAKE_LINK_SHLIB="%s"' % value)
  #            self.run("qmake %s" % " ".join(args), run_environment=True)
  #            if tools.os_info.is_windows:
  #                if self.settings.compiler == "Visual Studio":
  #                    self.run("nmake", run_environment=True)
  #                else:
  #                    self.run("mingw32-make", run_environment=True)
  #            else:
  #                self.run("make", run_environment=True)

  def _build_with_cmake_find_package_multi(self):
    #cmake = CMake(self, set_cmake_flags=True)
    cmake = CMake(self)
    #if self.settings.os == "Macos":
    #  cmake.definitions['CMAKE_OSX_DEPLOYMENT_TARGET'] = '10.14'
    cmake.configure()
    cmake.build()
    if not self.conf.get("tools.build:skip_test", default=False):
      if can_run(self):
        cmake.test()
    #cmake.build(target="package")

  def _test_with_qmake(self):
      if not self._qmake_supported():
          return
      self.output.info("Testing qmake")
      #bin_path = os.path.join("qmake_folder", "bin")
      #if tools.os_info.is_macos:
      #    bin_path = os.path.join(bin_path, "test_package.app", "Contents", "MacOS")
      #shutil.copy("qt.conf", bin_path)
      #self.run(os.path.join(bin_path, "test_package"), run_environment=True)

  def _test_with_cmake_find_package_multi(self):
      self.output.info("Testing CMake_find_package_multi")
      #shutil.copy("qt.conf", "bin")
      #self.run(os.path.join("bin", "test_package"), run_environment=True)

  def test(self):
      if not tools.cross_building(self.settings, skip_x64_x86=True):
          self._test_with_qmake()
          self._test_with_cmake_find_package_multi()

  def package_id(self):
    del self.info.options.python_executable  # PATH to the interpreter is not important, only version matters
    #if self.info.options.without_python:
    #    del self.info.options.python_version

  #def package(self):
  #    copy(
  #        self,
  #        "LICENSE.md",
  #        self.source_folder,
  #        os.path.join(self.package_folder, "licenses"),
  #    )
  #    cmake = CMake(self)
  #    cmake.install()
  #    rmdir(self, os.path.join(self.package_folder, "lib", "cmake"))
  #
  #def package_info(self):
  #    compiler = self.settings.compiler
  #    self.cpp_info.components["core"].requires = ["gsl-lite::gsl-lite"]
  #    if self.options.use_fmtlib:
  #        self.cpp_info.components["core"].requires.append("fmt::fmt")
  #    if compiler == "msvc":
  #        self.cpp_info.components["core"].cxxflags = ["/utf-8"]
  #    self.cpp_info.components["systems"].requires = ["core"]

-------- Installing package xkbcommon/1.5.0 (71 of 80) --------
xkbcommon/1.5.0: Building from source
xkbcommon/1.5.0: Package xkbcommon/1.5.0:85e159c0746b2a8d98e6564905f7a4459148bba6
xkbcommon/1.5.0: Copying sources to build folder
xkbcommon/1.5.0: Building your package in /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b
xkbcommon/1.5.0: Calling generate()
xkbcommon/1.5.0: Generators folder: /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release/conan
WARN: deprecated: Please, do not use a Conan option value directly. Convert 'options.with_wayland' into a valid Pythondata type, e.g, bool(self.options.shared)
WARN: deprecated: Please, do not use a Conan option value directly. Convert 'options.with_x11' into a valid Pythondata type, e.g, bool(self.options.shared)
WARN: deprecated: Please, do not use a Conan option value directly. Convert 'options.xkbregistry' into a valid Pythondata type, e.g, bool(self.options.shared)
xkbcommon/1.5.0: Generating aggregated env files
xkbcommon/1.5.0: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
xkbcommon/1.5.0: Calling build()
xkbcommon/1.5.0: Meson configure cmd: meson setup --native-file "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release/conan/conan_meson_native.ini" "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release" "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/src" -Dprefix="/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/p"
xkbcommon/1.5.0: RUN: meson setup --native-file "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release/conan/conan_meson_native.ini" "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release" "/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/src" -Dprefix="/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/p"
The Meson build system
Version: 1.3.1
Source dir: /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/src
Build dir: /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release
Build type: native build
Project name: libxkbcommon
Project version: 1.5.0
C compiler for the host machine: gcc (gcc 11.4.0 "gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0")
C linker for the host machine: gcc ld.bfd 2.38
Host machine cpu family: x86_64
Host machine cpu: x86_64
Compiler for C supports arguments -fno-strict-aliasing: YES
Compiler for C supports arguments -Wno-unused-parameter: YES
Compiler for C supports arguments -Wno-missing-field-initializers: YES
Compiler for C supports arguments -Wpointer-arith: YES
Compiler for C supports arguments -Wmissing-declarations: YES
Compiler for C supports arguments -Wformat=2: YES
Compiler for C supports arguments -Wstrict-prototypes: YES
Compiler for C supports arguments -Wmissing-prototypes: YES
Compiler for C supports arguments -Wnested-externs: YES
Compiler for C supports arguments -Wbad-function-cast: YES
Compiler for C supports arguments -Wshadow: YES
Compiler for C supports arguments -Wlogical-op: YES
Compiler for C supports arguments -Wdate-time: YES
Compiler for C supports arguments -Wwrite-strings: YES
Compiler for C supports arguments -Wno-documentation-deprecated-sync: NO
Found pkg-config: YES (/mnt/d/mc/1/3/.conan2/p/pkgco03f8fb03aa80a/p/bin/pkgconf) 2.1.0
Run-time dependency xkeyboard-config found: YES 2.33
Has header "unistd.h" : YES
Checking if "__builtin_expect" : links: YES
Header "unistd.h" has symbol "eaccess" : YES
Header "unistd.h" has symbol "euidaccess" : YES
Header "sys/mman.h" has symbol "mmap" : YES
Header "stdlib.h" has symbol "mkostemp" : YES
Header "fcntl.h" has symbol "posix_fallocate" : YES
Header "string.h" has symbol "strndup" : YES
Header "stdio.h" has symbol "asprintf" : YES
Header "stdlib.h" has symbol "secure_getenv" : YES
Header "limits.h" has symbol "PATH_MAX" : YES
Checking if "-Wl,--version-script" : links: YES
Program scripts/map-to-def found: YES (/mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/src/scripts/map-to-def)
Program bison found: YES (/mnt/d/mc/1/3/.conan2/p/b/bisona54b4b5f63d0b/p/bin/bison)
Run-time dependency xcb found: YES 1.14
Run-time dependency xcb-xkb found: YES 1.14
Run-time dependency libxml-2.0 found: YES 2.12.3
Header "getopt.h" has symbol "getopt_long" : YES
Has header "linux/input.h" : YES
Found CMake: /usr/bin/cmake (3.22.1)
Run-time dependency wayland-client found: NO (tried pkgconfig and cmake)
Run-time dependency wayland-protocols found: YES 1.32
Build-time dependency wayland-scanner_build found: YES 1.22.0

../src/meson.build:481:12: ERROR: Problem encountered: The Wayland xkbcli programs require wayland-client >= 1.2.0, wayland-protocols >= 1.7 which were not found.
You can disable the Wayland xkbcli programs with -Denable-wayland=false.

A full log can be found at /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release/meson-logs/meson-log.txt

xkbcommon/1.5.0: ERROR:
Package '85e159c0746b2a8d98e6564905f7a4459148bba6' build failed
xkbcommon/1.5.0: WARN: Build folder /mnt/d/mc/1/3/.conan2/p/b/xkbco3a809b8291313/b/build-release
ERROR: xkbcommon/1.5.0: Error in build() method, line 144
        meson.configure()
        ConanException: Error 1 while executing

memsharded commented 8 months ago

Thanks for the details.

I have tried it, had to change:

  def set_version(self):
    self.version = "1.0"

Then, after that, with the command:

conan install . -c tools.system.package_manager:mode=install -c tools.system.package_manager:sudo=True --build=missing -o "*:shared=True" -o "abseil/*:shared=False" -o "boost*:shared=True" -o qt*:shared=True -c tools.build:verbosity=verbose -c tools.build:jobs=4 -s compiler.cppstd=gnu17 -c tools.build:skip_test=True -s build_type=Debug

I had to quote the "*".

Then I got a normal conflict:

ERROR: Version conflict: wayland/1.22.0->expat/2.5.0, ->expat/2.6.0.

Not the error you are reporting inside the Meson build. Testing with latest Conan 2.1.

Then I used the new 2.1 to resolve the conflict:

[replace_requires]
expat/*: expat/2.6.0

and pass it both as host and build profile

Then it started to build things, but then I got stuck with:

libelf/0.8.13: WARN: network: Error downloading file http://repository.timesys.com/buildsources/l/libelf/libelf-0.8.13/libelf-0.8.13.tar.gz: 'HTTPConnectionPool(host='repository.timesys.com', port=80): Max retries exceeded with url: /buildsources/l/libelf/libelf-0.8.13/libelf-0.8.13.tar.gz (Caused by ResponseError('too many 504 error responses'))'

It seems their server is down at this moment

It eventually proceed. Then I wanted to introduce some prints/traces in the xkbcommon recipe, but then the latest revisions from ConanCenter brought a new version conflict, I added a new replace:

libxml2/*: libxml2/2.12.3

to be able to move forward.

I now think that I have been able to identify the issue in PkgConfigDeps, I will be working on a fix for Conan 2.2. Sorry it took so long, it was challenging to reproduce.

memsharded commented 8 months ago

Submitting a fix in https://github.com/conan-io/conan/pull/15763

The fix was quick once we could reproduce it. This highlights the importance of reducing to the very minimum the repro-cases, it helps a lot to expedite the resolution of bugs.

I have already tried to backport this to Conan 1.X, but it doesn't seem possible, as the graph model (tool_requires when you are already in the build context) in Conan 1.X does not correctly support this case.

Thanks all for the feedback!

memsharded commented 8 months ago

Closed by https://github.com/conan-io/conan/pull/15763, it will be in next 2.2

conan-io / conan