search

conan-io / conan-center-index

Recipes for the ConanCenter repository
https://conan.io/center
MIT License
921 stars 1.66k forks source link

[package] ffmpeg/6.1+others: pulls in libva-dev, for which there is no conan recipe, instead of libva #24228

Open aentwist opened 1 month ago

aentwist commented 1 month ago

Description

See https://stackoverflow.com/questions/78576626/how-does-conan-handle-dev-libraries

ffmpeg tries to install libva-dev, for which there is no conan recipe, and it fails.

Note that there is a recipe for libva. We should be using that. Conda succeeds here. It uses libva.

Package and Environment Details

Conan profile

[settings] arch=x86_64 build_type=Release compiler=gcc compiler.cppstd=gnu17 compiler.libcxx=libstdc++11 compiler.version=12 os=Linux

Steps to reproduce

conan profile detect --force
conan install conanfile.txt --build=missing

conanfile.txt

[requires]
ffmpeg/6.1

Logs

8 - Download (conancenter)
Skipped binaries
    meson/1.2.2
dpkg-query: no packages found matching libva-dev
ERROR: vaapi/system: Error in system_requirements() method, line 38
        apt.install(["libva-dev"], update=True, check=True)
        ConanException: System requirements: 'libva-dev' are missing but can't install because tools.system.package_manager:mode is 'check'.Please update packages manually or set 'tools.system.package_manager:mode' to 'install' in the [conf] section of the profile, or in the command line using '-c tools.system.package_manager:mode=install'
memsharded commented 1 month ago

Hi @aentwist

I am not sure if you have checked the error message:

ConanException: System requirements: 'libva-dev' are missing but can't install because tools.system.package_manager:mode is 'check'.Please update packages manually or set 'tools.system.package_manager:mode' to 'install' in the [conf] section of the profile, or in the command line using '-c tools.system.package_manager:mode=install'

That is, the system package is not installed but by default Conan will not automatically install system packages, it only does a check.

Can you please try again with:

conan install conanfile.txt --build=missing -c tools.system.package_manager:mode=install -c tools.system.package_manager:sudo=True

The second argument only if the system needs sudo to install system packages. Please try that and let us know the output.

aentwist commented 1 month ago

No, I don't have root access and cannot build containerized because I'm building desktop apps. The alternative to a package manager is compiling it manually

aentwist commented 1 month ago

I don't know what else to say about this besides my stackoverflow question and that I just did it with conda. Admittedly, I have no idea how they sub libva for libva-dev / make it work.

Edit: here is what I got,

memsharded commented 1 month ago

No, I don't have root access and cannot build containerized because I'm building desktop apps. The alternative to a package manager is compiling it manually

But many desktop apps also rely on system packages often, it is not that unusual to have to install something in the system.

I think this might be a candidate for the new [replace_requires], see https://blog.conan.io/2024/02/20/Conan-2-graph-features.html

The idea is that the Conan recipes in ConanCenter have a dependency to vaapi/system, to depend by default on the system package (there are challenges with some low level packages close to the system). This can be hot-swapped or replaced by users, specifying in their profile something like:

[replace_requires]
vaapi/system: libva/2.21.0
aentwist commented 1 month ago

I am too new to conan to understand what this means or entails. I can imagine there are large challenges with extra low c libraries. For completeness, the last of the information I have is exactly what conda decided to install that made this work. This is what I got from installing av, not ffmpeg so take it with a large grain of salt.

conda C deps 
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - aom=3.9.0=hac33072_0
  # - av=12.0.0=py312hb4c3d57_1
  - bzip2=1.0.8=hd590300_5
  - ca-certificates=2024.6.2=hbcca054_0
  - cairo=1.18.0=h3faef2a_0
  - dav1d=1.2.1=hd590300_0
  - expat=2.6.2=h59595ed_0
  - ffmpeg=6.1.1=gpl_he44c6f3_112
  - font-ttf-dejavu-sans-mono=2.37=hab24e00_0
  - font-ttf-inconsolata=3.000=h77eed37_0
  - font-ttf-source-code-pro=2.038=h77eed37_0
  - font-ttf-ubuntu=0.83=h77eed37_2
  - fontconfig=2.14.2=h14ed4e7_0
  - fonts-conda-ecosystem=1=0
  - fonts-conda-forge=1=0
  - freetype=2.12.1=h267a509_2
  - fribidi=1.0.10=h36c2ea0_0
  - gettext=0.22.5=h59595ed_2
  - gettext-tools=0.22.5=h59595ed_2
  - gmp=6.3.0=h59595ed_1
  - gnutls=3.7.9=hb077bed_0
  - graphite2=1.3.13=h59595ed_1003
  - harfbuzz=8.5.0=hfac3d4d_0
  - icu=73.2=h59595ed_0
  - lame=3.100=h166bdaf_1003
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.40=hf3520f5_1
  - lerc=4.0.0=h27087fc_0
  - libabseil=20240116.2=cxx17_h59595ed_0
  - libasprintf=0.22.5=h661eb56_2
  - libasprintf-devel=0.22.5=h661eb56_2
  - libass=0.17.1=h8fe9dca_1
  - libblas=3.9.0=22_linux64_openblas
  - libcblas=3.9.0=22_linux64_openblas
  - libdeflate=1.20=hd590300_0
  - libdrm=2.4.120=hd590300_0
  - libexpat=2.6.2=h59595ed_0
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=13.2.0=h77fa898_7
  - libgettextpo=0.22.5=h59595ed_2
  - libgettextpo-devel=0.22.5=h59595ed_2
  - libgfortran-ng=13.2.0=h69a702a_7
  - libgfortran5=13.2.0=hca663fb_7
  - libglib=2.80.2=hf974151_0
  - libgomp=13.2.0=h77fa898_7
  - libhwloc=2.10.0=default_h5622ce7_1001
  - libiconv=1.17=hd590300_2
  - libidn2=2.3.7=hd590300_0
  - libjpeg-turbo=3.0.0=hd590300_1
  - liblapack=3.9.0=22_linux64_openblas
  - libnsl=2.0.1=hd590300_0
  - libopenblas=0.3.27=pthreads_h413a1c8_0
  - libopenvino=2024.1.0=h2da1b83_7
  - libopenvino-auto-batch-plugin=2024.1.0=hb045406_7
  - libopenvino-auto-plugin=2024.1.0=hb045406_7
  - libopenvino-hetero-plugin=2024.1.0=h5c03a75_7
  - libopenvino-intel-cpu-plugin=2024.1.0=h2da1b83_7
  - libopenvino-intel-gpu-plugin=2024.1.0=h2da1b83_7
  - libopenvino-intel-npu-plugin=2024.1.0=he02047a_7
  - libopenvino-ir-frontend=2024.1.0=h5c03a75_7
  - libopenvino-onnx-frontend=2024.1.0=h07e8aee_7
  - libopenvino-paddle-frontend=2024.1.0=h07e8aee_7
  - libopenvino-pytorch-frontend=2024.1.0=he02047a_7
  - libopenvino-tensorflow-frontend=2024.1.0=h39126c6_7
  - libopenvino-tensorflow-lite-frontend=2024.1.0=he02047a_7
  - libopus=1.3.1=h7f98852_1
  - libpciaccess=0.18=hd590300_0
  - libpng=1.6.43=h2797004_0
  - libprotobuf=4.25.3=h08a7969_0
  - libsqlite=3.45.3=h2797004_0
  - libstdcxx-ng=13.2.0=hc0a3c3a_7
  - libtasn1=4.19.0=h166bdaf_0
  - libtiff=4.6.0=h1dd3fc0_3
  - libunistring=0.9.10=h7f98852_0
  - libuuid=2.38.1=h0b41bf4_0
  - libva=2.21.0=hd590300_0  # <- not libva-dev
  - libvpx=1.14.0=h59595ed_0
  - libwebp-base=1.4.0=hd590300_0
  - libxcb=1.15=h0b41bf4_0
  - libxcrypt=4.4.36=hd590300_1
  - libxml2=2.12.7=hc051c1a_0
  - libzlib=1.3.1=h4ab18f5_1
  - ncurses=6.5=h59595ed_0
  - nettle=3.9.1=h7ab15ed_0
  - numpy=1.26.4=py312heda63a1_0
  - ocl-icd=2.3.2=hd590300_1
  - openh264=2.4.1=h59595ed_0
  - openjpeg=2.5.2=h488ebb8_0
  - openssl=3.3.0=h4ab18f5_3
  - p11-kit=0.24.1=hc5aa10d_0
  - pcre2=10.43=hcad00b1_0
  - pillow=10.3.0=py312hdcec9eb_0
  - pip=24.0=pyhd8ed1ab_0
  - pixman=0.43.2=h59595ed_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pugixml=1.14=h59595ed_0
  - python=3.12.3=hab00c5b_0_cpython
  - python_abi=3.12=4_cp312
  - readline=8.2=h8228510_1
  - setuptools=70.0.0=pyhd8ed1ab_0
  - snappy=1.2.0=hdb0a2a9_1
  - svt-av1=2.1.0=hac33072_0
  - tbb=2021.12.0=h297d8ca_1
  - tk=8.6.13=noxft_h4845f30_101
  - tzdata=2024a=h0c530f3_0
  - wheel=0.43.0=pyhd8ed1ab_1
  - x264=1!164.3095=h166bdaf_2
  - x265=3.5=h924138e_3
  - xorg-fixesproto=5.0=h7f98852_1002
  - xorg-kbproto=1.0.7=h7f98852_1002
  - xorg-libice=1.1.1=hd590300_0
  - xorg-libsm=1.2.4=h7391055_0
  - xorg-libx11=1.8.9=h8ee46fc_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xorg-libxext=1.3.4=h0b41bf4_2
  - xorg-libxfixes=5.0.3=h7f98852_1004
  - xorg-libxrender=0.9.11=hd590300_0
  - xorg-renderproto=0.11.1=h7f98852_1002
  - xorg-xextproto=7.3.0=h0b41bf4_1003
  - xorg-xproto=7.0.31=h7f98852_1007
  - xz=5.2.6=h166bdaf_0
  - zlib=1.3.1=h4ab18f5_1
  - zstd=1.5.6=ha6fb4c9_0
aentwist commented 1 month ago

If there is a conclusive answer as to

It would give me peace of mind, if nothing else. Anyways, I'll leave things to you. I initially thought this was a solved problem in conan and subbing libva for libva-dev would be somewhat well-defined and trivial, as libva-dev has no conan recipe.

memsharded commented 1 month ago

whether *-dev packages are in-scope

I am not sure what you mean. -dev packages are a Linux/Debian thing, nothing to do with Conan packages. It is true that some ConanCenter packages might conditionally depend on Linux/Debian on system packages and express in their recipes that this dependency is necessary, but those are no longer Conan packages but pre-requisites in the system. Conan does not model them in ConanCenter.

Please note that Conan aims to be very distributed and ConanCenter is a convenient resource, but might not cover all cases. There are many users that customize the ConanCenter recipes, forking the repo and customizing the recipes. Note there are features like the conan-center-index that allow to more easily use forks from conan-center-index, see https://blog.conan.io/2024/04/23/Introducing-local-recipes-index-remote.html. So one thing is Conan and its usage, and another very different thing is ConanCenter. The fact that recipes in ConanCenter implement one approach doesn't mean that that is the only possible approach.

I am too new to conan to understand what this means or entails.

It means that the recipes in ConanCenter default to use the system package libva-dev, which is good for the majority of cases.

Yours seems to be a more advance case, which Conan is aiming to support via the [replace_requires] feature.

as libva-dev has no conan recipe.

But there is a libva/2.21.0 recipe, which builds that library? So there is a Conan recipe for it. I am not sure if I understand.

uilianries commented 4 weeks ago

@aentwist Thank you for yourt information. I also would like to bring your comment about not being possible to use containers: https://stackoverflow.com/questions/78576626/how-does-conan-handle-dev-libraries#comment138534698_78576626

Some Conan package in Conan Center are wrappers to system packages because they are or really hard to be packaged, or very close to kernel/hardware/driver and building from source will probably not work for users when consuming.

About the ffmpeg Conan package, it does not consume libva, but vaapi/system (intel-vaapi-driver). You check all requirements in the Conan Center page (https://conan.io/center/recipes/ffmpeg?version=6.1) or by running the command conan graph info --requires=ffmpeg/6.1

Still, it requires xorg and vdpau from system. In Conda is true they have packaged and built them from source, also they have xorg distributed as components (IIRC there is a discussion about splitting xorg in Conan Center).

As workaround, you could install ffmpeg without those dependencies, in case they are not required. Otherwise, you will need to use from your system.

[requires]
ffmpeg/6.1

[options]
ffmpeg/*:with_vaapi=False
ffmpeg/*:with_vdpau=False
ffmpeg/*:with_xcb=False

As this configuration is not available in Conan Center for pre-built packages, you will need to build from source: conan install conanfile.txt --build=missing

aentwist commented 4 weeks ago

Thanks @uilianries, that is very informative. I'm excited to watch how things grow as the package ecosystem matures and less packages are placeholders for system ones!

About libva-dev, maybe I've had a fundamental misunderstanding. I thought libva-dev was explicitly specified in some recipe, like how conda specifies libva. But maybe it isn't specified at all, and things just automatically try to grab it. I am sorry for not conveying this key assumption - I did not expect there to be placeholder packages - they hide the fact that conan does not have a particular package. I see now that memsharded tried to tell me this multiple times, but none of them hit, maybe I was having a day yesterday. From this perspective @memsharded suggestion of replace_requires makes a ton more sense. Hm.

So the idea would then be that the error about libva-dev is caused because vaapi requires that, so we replace vaapi. Interesting. Indeed, with this helpful graph command conan graph info --requires=ffmpeg/6.1 | grep libva I see that libva-dev is not specified anywhere, so it is somehow an indirect dependency (I guess the graph shows direct dependencies only, or vaapi is a placeholder package for system ones, which depend on libva-dev). Nice.

If you don't think there are any actionable items to come out of this issue other than the vague idea of wider, less system-reliant package support feel free to close it. I hope it helps someone.