[question] Package with the tool requirement.

damian-tomczak commented 1 year ago

What is your question?

Hi,

I struggling with how to handle makefile package that requires tool built from the source.

So ./configure of this package checks if on the system is installed tool - exactly this tool: https://github.com/libbpf/bpftool.git

but I have no idea how to handle it. Should I create another package if so have the conanfile.py of this package should look like?

If something is misunderstand please me know.

Best Regards Thanks in advance.

Have you read the CONTRIBUTING guide?

[X] I've read the CONTRIBUTING guide

memsharded commented 1 year ago

Hi @damian-tomczak

If you need bpftool in your machine, you have several options:

Install it with the system package manager if it is there. Use the system_requirements() method of your recipe to call apt or other system package manager to get it.
Build it yourself, manually or automatically and consider it in the machine the moment your package run (part of provisioning your machines)
Create a Conan package to wrap it and use it as tool_requires.

For creating a recipe for a tool_requires please have a look: https://docs.conan.io/2/tutorial/creating_packages/other_types_of_packages/tool_requires_packages.html

The specific for building every project depends on the project, its build system, its dependencies (it might be necessary to create other packages for them, or vendorize them in the current package). There is no single solution for this, it really depends on every project.

damian-tomczak commented 1 year ago

Hi @damian-tomczak

If you need bpftool in your machine, you have several options:

Install it with the system package manager if it is there. Use the system_requirements() method of your recipe to call apt or other system package manager to get it.

Build it yourself, manually or automatically and consider it in the machine the moment your package run (part of provisioning your machines)

Create a Conan package to wrap it and use it as tool_requires.

For creating a recipe for a tool_requires please have a look: https://docs.conan.io/2/tutorial/creating_packages/other_types_of_packages/tool_requires_packages.html

The specific for building every project depends on the project, its build system, its dependencies (it might be necessary to create other packages for them, or vendorize them in the current package). There is no single solution for this, it really depends on every project.

Hi @memsharded

Okay tool_requires takes the bin to the package. But then I want to use this in Makefile that I don't want to change and it expects to find this bin in $PATH instead of somewhere in the package.

Are you getting my problem? Do I should just copy it to the /usr/bin? or does conan provide a nice way to do it?

memsharded commented 1 year ago

Are you getting my problem? Do I should just copy it to the /usr/bin? or does conan provide a nice way to do it?

Yes, definitely do NOT copy things to /usr/bin. That would be polluting your system and causing lots of troubles, conflicts between versions, etc.

The recommended way is:

If you put things in a package in the "bin" folder (the default of self.cppinfo.bindirs), that folder will be automatically added to the VirtualBuildEnv PATH env-var
Any consumer running self.run(cmd), (that defaults internally to self.run(cmd, env="conanbuild"), will automatically inject the "bin" folder of tool_requires into the environment $PATH variable before running the command.
And then, it will be automatically cleaned after the command is run, not polluting the environment or your system

So in short, this is fully automatic by Conan, no need to do anything special, and extremely discouraged to copy things to system paths. You can read more about using tool_requires in https://docs.conan.io/2/tutorial/consuming_packages/use_tools_as_conan_packages.html

damian-tomczak commented 1 year ago

Czy rozumiesz mój problem? Czy powinienem po prostu skopiować go do /usr/bin? czy też conan zapewnia dobry sposób na zrobienie tego?

Tak, zdecydowanie NIE kopiuj rzeczy do /usr/bin. To zanieczyszczałoby twój system i powodowało wiele problemów, konfliktów między wersjami itp.

Zalecany sposób to:

Jeśli umieścisz rzeczy w paczce w folderze „bin” (domyślnie self.cppinfo.bindirs), ten folder zostanie automatycznie dodany do VirtualBuildEnvPATH env-var

Każdy konsument, który uruchomi self.run(cmd), (który domyślnie ma wewnętrznie wartość self.run(cmd, env="conanbuild"), automatycznie wstawi folder „bin” tool_requiresdo zmiennej środowiskowej $PATH przed uruchomieniem polecenia.

A następnie zostanie automatycznie wyczyszczony po uruchomieniu polecenia, nie zanieczyszczając środowiska ani systemu

Krótko mówiąc, jest to w pełni automatyczne przez Conana, nie trzeba robić nic specjalnego i bardzo zniechęca do kopiowania rzeczy do ścieżek systemowych. Możesz przeczytać więcej o używaniu tool_requiresw https://docs.conan.io/2/tutorial/using_packages/use_tools_as_conan_packages.html

Okay thanks, one more thing:

    def build_requirements(self):
        self.tool_requires("bpftool/v7.2.0")

    def generate(self):
        print(1234)
        for require, value in self.dependencies.items():
            if require.ref.name == "bpftool":
                print(require, value)
        print(1234)
        print(self.dependencies['bpftool'].cppinfo.bindirs)
        tc = AutotoolsToolchain(self)
        tc.generate()

ERROR: xdp-tools/v1.4.0: Error in generate() method, line 24
        print(self.dependencies['bpftool'].cppinfo.bindirs)
        KeyError: "'bpftool' not found in the dependency set"

problem doesnt't occur if bpftool is defined as requires instead of tool_requires.

Can you help?

memsharded commented 1 year ago

Yes, you need to specify self.dependencies.build['bpftool'].cppinfo.bindirs, you can find more info about this in https://docs.conan.io/2/reference/conanfile/methods/generate.html#conan-conanfile-model-dependencies

damian-tomczak commented 1 year ago

Yes, you need to specify self.dependencies.build['bpftool'].cppinfo.bindirs, you can find more info about this in https://docs.conan.io/2/reference/conanfile/methods/generate.html#conan-conanfile-model-dependencies

Could you help me, I have no idea what I'm doing wrong:

from conan import ConanFile
from conan.tools.gnu import Autotools, AutotoolsToolchain
from conan.tools.scm import Git
import os

class BpfTool(ConanFile):
    name = "bpftool"
    settings = "os", "arch", "compiler", "build_type"

    def source(self):
        git = Git(self)
        git.clone(url="https://github.com/libbpf/bpftool", target=".", args=["--recursive"])
        git.checkout(self.version)

    def generate(self):
        tc = AutotoolsToolchain(self)
        tc.generate()

    def build(self):
        os.chdir(os.path.join('src'))
        autotools = Autotools(self)
        autotools.make()

    def package(self):
        os.chdir(os.path.join('src'))
        autotools = Autotools(self)
        autotools.install()

    def package_info(self):
        self.cpp_info.bindirs = ["usr/local/sbin"]

from conan import ConanFile
from conan.tools.gnu import Autotools, AutotoolsToolchain
from conan.tools.scm import Git
import os
import shutil

class XdpTools(ConanFile):
    name = "xdp-tools"
    settings = "os", "arch", "compiler", "build_type"
    args = ["BUILD_STATIC_ONLY=1"]
    generators = "AutotoolsDeps"

    def source(self):
        git = Git(self)
        git.clone(url="https://github.com/xdp-project/xdp-tools", target=".", args=["--recursive"])
        git.checkout(self.version)

    def generate(self):
        tc = AutotoolsToolchain(self)
        tc.generate()

    def build_requirements(self):
        self.test_requires("bpftool/v7.2.0")

    def build(self):
        self.run("bpftool")
        autotools = Autotools(self)
        autotools.configure(args=self.args)
        autotools.make(args=self.args)

root@VN1411-ubuntu:~/.conan2/p/b/bpftoc1c1b5d2c5c73/p# tree
.
|-- conaninfo.txt
|-- conanmanifest.txt
`-- usr
    |-- local
    |   `-- sbin
    |       `-- bpftool
    `-- share
        `-- bash-completion
            `-- completions
                `-- bpftool

6 directories, 4 files

damian-tomczak commented 1 year ago

@memsharded

xdp-tools/v1.4.0: Calling build()
xdp-tools/v1.4.0: RUN: bpftool
/bin/sh: 1: bpftool: not found

damian-tomczak commented 1 year ago

@memsharded

I think that something may be wrong with conanautotoolsdeps.sh

script_folder="/root/.conan2/p/b/xdp-t477a962f685e4/b"
echo "echo Restoring environment" > "$script_folder/deactivate_conanautotoolsdeps.sh"
for v in CPPFLAGS LIBS LDFLAGS CXXFLAGS CFLAGS
do
    is_defined="true"
    value=$(printenv $v) || is_defined="" || true
    if [ -n "$value" ] || [ -n "$is_defined" ]
    then
        echo export "$v='$value'" >> "$script_folder/deactivate_conanautotoolsdeps.sh"
    else
        echo unset $v >> "$script_folder/deactivate_conanautotoolsdeps.sh"
    fi
done

export CPPFLAGS="$CPPFLAGS"
export LIBS="$LIBS"
export LDFLAGS="$LDFLAGS"
export CXXFLAGS="$CXXFLAGS"
export CFLAGS="$CFLAGS"

memsharded commented 1 year ago

But you have changed and doing self.test_requires("bpftool/v7.2.0"), that seems incorrect, it is an executable, you need to do self.tool_requires("bpftool/v7.2.0")

memsharded commented 1 year ago

generators = "AutotoolsDeps" is useless if you only depend on a tool_requires, you can drop it. That is only used for regular library requires

damian-tomczak commented 1 year ago

self.tool_requires("bpftool/v7.2.0")

OMGGGGGGGG, THANK YOU I'M COMPLETELY BLIND

memsharded commented 1 year ago

You had it right in your first comment https://github.com/conan-io/conan/issues/14286#issuecomment-1637536126 😄

conan-io / conan

[question] Package with the tool requirement. #14286

What is your question?

Have you read the CONTRIBUTING guide?