AlexRamallo commented 1 year ago

What is your question?

Hi,

I'm not sure if this is a bug with Conan or a bug with my generator. Basically, my waf generator was working fine until I updated Conan recently. After doing a bisect, I found the exact change that broke it, but I'm not certain why. The relevant PR is #14673, and the exact line that breaks it is here. Commenting that line out fixes the problem for me (tested on the current develop2 branch).

Problem description

The core issue seems to be that, in some circumstances (specifically, when installing a shared library from the cache), the "components" list isn't being populated for some transient dependencies. Basically, there's a section in my generate() function that iterates packages like this:

for req, dep in self.conanfile.dependencies.items():
    if dep.cpp_info.has_components:
        #...
    else:
        #...

However, has_components returns false sometimes even when it should have components. This breaks my generator implementation when a package depends on one of those components.

This seems like a Conan bug to me, but the CMake generators dont have in issue here. It could just be that I'm doing something wrong in my generator.

Reproduce

The generator repo has a test project called tests/test_sdl which reproduces the issue (tested on Fedora 38 and Ubuntu 22.04).

git clone https://github.com/AlexRamallo/waf-conan-generator
cd waf-conan-generator

#export generator package
conan export . 

#try to install dependencies
cd tests/test_sdl
conan install . -of=build --build=missing

#if it succeeded, call install AGAIN
conan install . -of=build --build=missing

It will succeed if conan builds the package (e.g. it's not cached). Otherwise, if the package exists in the cache, then it will fail. It also doesn't seem to happen when building with shared=False for SDL.

The failure happens because the dependency wayland doesn't report any components even though it should, and the package xkbcommon explicitly requires one of those components. My generator needs to perform a topological sort on all of the packages to ensure proper link order, and it assumes that all referenced components/packages are valid. I'm not sure if that's the right approach or not, so please let me know if there's a better way.

Logs

I added a bunch of print statements to the generator, and the problem is clearly visible:

Successful install (SDL isn't in the cache when conan install is called):

``` ======== Finalizing install (deploy, generators) ======== conanfile.py (waf_conan_test_sdl/1.0): Calling generate() conanfile.py (waf_conan_test_sdl/1.0): Generators folder: /waf-conan-generator/tests/test_sdl/build sdl ver: 2.28.3 comp: libsdl2 => use_name: sdl_libsdl2 comp: sdl2main => use_name: sdl_sdl2main parent use_name: sdl pulseaudio ver: 14.2 comp: pulse => use_name: pulseaudio_pulse comp: pulse-simple => use_name: pulseaudio_pulse_simple parent use_name: pulseaudio libsndfile ver: 1.2.2 comp: sndfile => use_name: libsndfile_sndfile parent use_name: libsndfile vorbis ver: 1.3.7 comp: vorbismain => use_name: vorbis_vorbismain comp: vorbisenc => use_name: vorbis_vorbisenc comp: vorbisfile => use_name: vorbis_vorbisfile comp: vorbisenc-alias => use_name: vorbis_vorbisenc_alias comp: vorbisfile-alias => use_name: vorbis_vorbisfile_alias parent use_name: vorbis flac ver: 1.4.2 comp: libflac => use_name: flac_libflac comp: libflac++ => use_name: flac_libflac++ parent use_name: flac ogg ver: 1.3.5 comp: ogglib => use_name: ogg_ogglib parent use_name: ogg opus ver: 1.4 comp: libopus => use_name: opus_libopus parent use_name: opus mpg123 ver: 1.31.2 comp: libmpg123 => use_name: mpg123_libmpg123 comp: libout123 => use_name: mpg123_libout123 comp: libsyn123 => use_name: mpg123_libsyn123 parent use_name: mpg123 libmp3lame ver: 3.100 libcap ver: 2.69 comp: cap => use_name: libcap_cap parent use_name: libcap libtool ver: 2.4.7 automake ver: 1.16.5 autoconf ver: 2.71 m4 ver: 1.4.19 libalsa ver: 1.2.10 openssl ver: 3.1.3 comp: crypto => use_name: openssl_crypto comp: ssl => use_name: openssl_ssl parent use_name: openssl opengl ver: system xkbcommon ver: 1.4.1 comp: libxkbcommon => use_name: xkbcommon_libxkbcommon comp: libxkbcommon-x11 => use_name: xkbcommon_libxkbcommon_x11 comp: libxkbregistry => use_name: xkbcommon_libxkbregistry comp: xkbcli-interactive-wayland => use_name: xkbcommon_xkbcli_interactive_wayland parent use_name: xkbcommon xkeyboard-config ver: system xorg ver: system comp: x11 => use_name: xorg_x11 comp: x11-xcb => use_name: xorg_x11_xcb comp: fontenc => use_name: xorg_fontenc comp: ice => use_name: xorg_ice comp: xau => use_name: xorg_xau comp: xaw7 => use_name: xorg_xaw7 comp: xcomposite => use_name: xorg_xcomposite comp: xcursor => use_name: xorg_xcursor comp: xdamage => use_name: xorg_xdamage comp: xdmcp => use_name: xorg_xdmcp comp: xext => use_name: xorg_xext comp: xfixes => use_name: xorg_xfixes comp: xi => use_name: xorg_xi comp: xinerama => use_name: xorg_xinerama comp: xkbfile => use_name: xorg_xkbfile comp: xmu => use_name: xorg_xmu comp: xmuu => use_name: xorg_xmuu comp: xpm => use_name: xorg_xpm comp: xrandr => use_name: xorg_xrandr comp: xrender => use_name: xorg_xrender comp: xres => use_name: xorg_xres comp: xscrnsaver => use_name: xorg_xscrnsaver comp: xt => use_name: xorg_xt comp: xtst => use_name: xorg_xtst comp: xv => use_name: xorg_xv comp: xvmc => use_name: xorg_xvmc comp: xxf86vm => use_name: xorg_xxf86vm comp: xcb-xkb => use_name: xorg_xcb_xkb comp: xcb-icccm => use_name: xorg_xcb_icccm comp: xcb-image => use_name: xorg_xcb_image comp: xcb-keysyms => use_name: xorg_xcb_keysyms comp: xcb-randr => use_name: xorg_xcb_randr comp: xcb-render => use_name: xorg_xcb_render comp: xcb-renderutil => use_name: xorg_xcb_renderutil comp: xcb-shape => use_name: xorg_xcb_shape comp: xcb-shm => use_name: xorg_xcb_shm comp: xcb-sync => use_name: xorg_xcb_sync comp: xcb-xfixes => use_name: xorg_xcb_xfixes comp: xcb-xinerama => use_name: xorg_xcb_xinerama comp: xcb => use_name: xorg_xcb comp: xcb-atom => use_name: xorg_xcb_atom comp: xcb-aux => use_name: xorg_xcb_aux comp: xcb-event => use_name: xorg_xcb_event comp: xcb-util => use_name: xorg_xcb_util comp: xcb-dri3 => use_name: xorg_xcb_dri3 comp: xcb-cursor => use_name: xorg_xcb_cursor comp: uuid => use_name: xorg_uuid comp: sm => use_name: xorg_sm parent use_name: xorg wayland ver: 1.22.0 comp: wayland-scanner => use_name: wayland_wayland_scanner comp: wayland-server => use_name: wayland_wayland_server comp: wayland-client => use_name: wayland_wayland_client comp: wayland-cursor => use_name: wayland_wayland_cursor comp: wayland-egl => use_name: wayland_wayland_egl comp: wayland-egl-backend => use_name: wayland_wayland_egl_backend parent use_name: wayland libffi ver: 3.4.4 libxml2 ver: 2.11.4 libiconv ver: 1.17 expat ver: 2.5.0 egl ver: system libunwind ver: 1.6.2 comp: unwind => use_name: libunwind_unwind comp: generic => use_name: libunwind_generic comp: ptrace => use_name: libunwind_ptrace comp: setjmp => use_name: libunwind_setjmp comp: coredump => use_name: libunwind_coredump parent use_name: libunwind xz_utils ver: 5.4.4 zlib ver: 1.3 conanfile.py (waf_conan_test_sdl/1.0): Generating aggregated env files conanfile.py (waf_conan_test_sdl/1.0): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] Install finished successfully ```

Notice that under "wayland" it lists all of its components successfully.

Here is the output when SDL exists in the cache at the time conan install is called:

``` ======== Finalizing install (deploy, generators) ======== conanfile.py (waf_conan_test_sdl/1.0): Calling generate() conanfile.py (waf_conan_test_sdl/1.0): Generators folder: /waf-conan-generator/tests/test_sdl/build sdl ver: 2.28.3 comp: libsdl2 => use_name: sdl_libsdl2 comp: sdl2main => use_name: sdl_sdl2main parent use_name: sdl pulseaudio ver: 14.2 comp: pulse => use_name: pulseaudio_pulse comp: pulse-simple => use_name: pulseaudio_pulse_simple parent use_name: pulseaudio libsndfile ver: 1.2.2 vorbis ver: 1.3.7 flac ver: 1.4.2 ogg ver: 1.3.5 opus ver: 1.4 mpg123 ver: 1.31.2 libmp3lame ver: 3.100 libcap ver: 2.69 libtool ver: 2.4.7 automake ver: 1.16.5 autoconf ver: 2.71 m4 ver: 1.4.19 libalsa ver: 1.2.10 openssl ver: 3.1.3 opengl ver: system xkbcommon ver: 1.4.1 comp: libxkbcommon => use_name: xkbcommon_libxkbcommon comp: libxkbcommon-x11 => use_name: xkbcommon_libxkbcommon_x11 comp: libxkbregistry => use_name: xkbcommon_libxkbregistry comp: xkbcli-interactive-wayland => use_name: xkbcommon_xkbcli_interactive_wayland parent use_name: xkbcommon xkeyboard-config ver: system xorg ver: system comp: x11 => use_name: xorg_x11 comp: x11-xcb => use_name: xorg_x11_xcb comp: fontenc => use_name: xorg_fontenc comp: ice => use_name: xorg_ice comp: xau => use_name: xorg_xau comp: xaw7 => use_name: xorg_xaw7 comp: xcomposite => use_name: xorg_xcomposite comp: xcursor => use_name: xorg_xcursor comp: xdamage => use_name: xorg_xdamage comp: xdmcp => use_name: xorg_xdmcp comp: xext => use_name: xorg_xext comp: xfixes => use_name: xorg_xfixes comp: xi => use_name: xorg_xi comp: xinerama => use_name: xorg_xinerama comp: xkbfile => use_name: xorg_xkbfile comp: xmu => use_name: xorg_xmu comp: xmuu => use_name: xorg_xmuu comp: xpm => use_name: xorg_xpm comp: xrandr => use_name: xorg_xrandr comp: xrender => use_name: xorg_xrender comp: xres => use_name: xorg_xres comp: xscrnsaver => use_name: xorg_xscrnsaver comp: xt => use_name: xorg_xt comp: xtst => use_name: xorg_xtst comp: xv => use_name: xorg_xv comp: xvmc => use_name: xorg_xvmc comp: xxf86vm => use_name: xorg_xxf86vm comp: xcb-xkb => use_name: xorg_xcb_xkb comp: xcb-icccm => use_name: xorg_xcb_icccm comp: xcb-image => use_name: xorg_xcb_image comp: xcb-keysyms => use_name: xorg_xcb_keysyms comp: xcb-randr => use_name: xorg_xcb_randr comp: xcb-render => use_name: xorg_xcb_render comp: xcb-renderutil => use_name: xorg_xcb_renderutil comp: xcb-shape => use_name: xorg_xcb_shape comp: xcb-shm => use_name: xorg_xcb_shm comp: xcb-sync => use_name: xorg_xcb_sync comp: xcb-xfixes => use_name: xorg_xcb_xfixes comp: xcb-xinerama => use_name: xorg_xcb_xinerama comp: xcb => use_name: xorg_xcb comp: xcb-atom => use_name: xorg_xcb_atom comp: xcb-aux => use_name: xorg_xcb_aux comp: xcb-event => use_name: xorg_xcb_event comp: xcb-util => use_name: xorg_xcb_util comp: xcb-dri3 => use_name: xorg_xcb_dri3 comp: xcb-cursor => use_name: xorg_xcb_cursor comp: uuid => use_name: xorg_uuid comp: sm => use_name: xorg_sm parent use_name: xorg wayland ver: 1.22.0 libffi ver: 3.4.4 libxml2 ver: 2.11.4 libiconv ver: 1.17 expat ver: 2.5.0 egl ver: system libunwind ver: 1.6.2 xz_utils ver: 5.4.4 zlib ver: 1.3 ERROR: conanfile.py (waf_conan_test_sdl/1.0): Error in generate() method, line 23 dep.generate() AssertionError: The following dependency for 'xkbcommon_xkbcli_interactive_wayland' wasn't found: 'wayland_wayland_client' is the package broken, or am I broken? ```

Notice how this time, there are no components under "wayland".

Thanks!

Have you read the CONTRIBUTING guide?

[X] I've read the CONTRIBUTING guide

memsharded commented 1 year ago

Hi @AlexRamallo

Thanks for your report. I have had a look. I think what is happening:

The first time, SDL needs to be built, and wayland is being used as a dependency (necessary at build time of SDL)
The second time, SDL is already built. The wayland dependency is no longer needed, and marked as skipped
Your generator is visiting absolutely all dependencies with self.conanfile.dependencies.items(), including those that need to be skipped, but you are not filtering those.
Skipped dependencies do not execute their package_info(), so they don't define components.

I think this would be solved with:

Iterate self.conanfile.dependencies.host.items() instead, to generate information for "host" requirements
The "build" requirements are typically iterated differently, via self.conanfile.dependencies.direct_build.items() to bring the executable in the PATH somehow.

Please try that and let us know, thanks!

AlexRamallo commented 1 year ago

Thanks for looking into this, that really helped me understand what was happening!

After switching to use dependencies.host, I ran into the same issue, except with libiconv. That one confused me because libiconv seemed to be a direct host dependency of sdl, yet it was missing too. So as a sanity check, I compared it to the output of CMakeDeps, and it too was missing the libiconv package, so I guess that's normal.

I removed the assertion and now just ignore missing packages/components since that seems to be the intended behavior. I still need to manually test some projects that depend on this to make sure nothing broke, but so far everything seem to be working.

memsharded commented 1 year ago

Thanks for the feedback, the changes look good! It is true that the dependency graph can become non-obvious, given the different features, like the binary skipping.

As a side note, I think it could be good for other users to contribute the Waf generator to the conan-extensions repo. It doesn't sound so mainstream to be built-in, but just in case some other user is using Waf, having a starting point could be great. I understand there are many situations it is not possible given company policies and such, but just in case 🙂

AlexRamallo commented 1 year ago

Sure there's no problem contributing this, but this code base was pulled out of another system with narrower requirements, so it might not work too well yet as a standalone generator. I'll try to add a few more tests and clean things up, then send a PR :)

memsharded commented 1 year ago

Sure there's no problem contributing this, but this code base was pulled out of another system with narrower requirements, so it might not work too well yet as a standalone generator. I'll try to add a few more tests and clean things up, then send a PR :)

Sure, no commitments, no rush, this is why conan-extensions is a better fit, it can have some warnings about the maturity, but still having something to start with can be better than having nothing

conan-io / conan

[question] (bug?) components from transient deps missing in generate() when loading from cache #14993

What is your question?

Problem description

Reproduce

Logs

Have you read the CONTRIBUTING guide?