Create a dummy libxcb-xlib.so.0

[gqmelo]

This library used to exist, but was deprecated. On older systems where it still exists (e.g SUSE 11), we may end up mixing the system's libxcb-xlib with the libX11 and libxcb provided by conda. This can cause missing symbols when trying to run a binary which links to libxcb.

NOTE:

If you are facing this problem you also need to install conda-forge's libx11. libx11 was the only real dependent of libxcb-xlib and was modified to acommodate the libxcb-xlib removal:

https://lists.freedesktop.org/archives/xcb/2009-April/004489.html https://lists.freedesktop.org/archives/xcb/2009-April/004492.html

[conda-forge-linter]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipe) and found it was in an excellent condition.

[gqmelo]

How to reproduce on Suse 11:

conda create -n test_xcb python=3.5 pyqt=5
source activate test_xcb
python -c "from PyQt5.QtWidgets import QApplication"

Output:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/lib64/libxcb-xlib.so.0: undefined symbol: _xcb_unlock_io

After the fix presented here I still get some error, now on system's libX11:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/lib64/libX11.so.6: undefined symbol: xcb_xlib_lock

But installing xorg-libx11 from conda-forge solves it:

conda install xorg-libx11
python -c "from PyQt5.QtWidgets import QApplication"

[ccordoba12]

@mingwandroid, what do you say?

[pkgw]

Hmmm, I'm having trouble understanding the problem case. Is it the case that the issue arises when you have conda-forge's libxcb installed, but not its libX11? If so, does the problem occur if you install the conda-forge libX11 as well?

[gqmelo]

@pkgw using an old libX11 with a more recent libxcb is one of the problems.

Using conda-forge's libX11 should fix that, however on SUSE11 there is the same problem the guy on the mailing list reported. libxcb-xlib was only used by libX11 and therefore nobody else should link to it, but because of how libtool works, lots of libraries which link to libX11 and libxcb are also linking to libxcb-xlib. Executing this on SUSE11 prints several libraries:

for f in `find /usr/lib64/ -name "*.so.*"`; do objdump -x $f | grep NEEDED | grep libxcb-xlib > /dev/null  && echo $f; done

Particularly the libraries I'm using are libGL.so.1 and libGLU.so.1, which are also wrongly linked to libxcb-xlib.so.0. That is why installing libX11 from conda-forge is not enough. If I load libGL.so.1 it will bring system's libxcb-xlib.so.0 which then depends on a symbol only present on system's libxcb.so.1

[pkgw]

Ah, I see. Well, that's all pretty gross, but given the circumstances this PR does seem like the best solution. The joys of Linux packaging!

[gqmelo]

Ah, I see. Well, that's all pretty gross, but given the circumstances this PR does seem like the best solution. The joys of Linux packaging!

Yeah, not proud of that solution :P. The real fix would be those still supported distros to rebuild all packages that should not link to libxcb-xlib.so.0, but I don't believe this will happen.

And just to add: the workaround presented here fix one of the problems: system libraries which are wrongly linking to libxcb-xlib.so.0. For a complete fix conda-forge's libX11 should be installed alongside libxcb.

Probably Qt5 should declare libX11 as dependency, but I'm afraid that could break on other systems. The experience we had at work is that if you use your own libX11 you should avoid using any other xorg libraries from the system, which means you have to provide all xorg libraries. There are some xorgs libraries that (not sure if intentionally) depends on libX11 private symbols and then they can complain about missing symbols when changing libX11 version. So Qt5 would have to bring all xorg libraries which maybe is too large a change.

[pkgw]

I totally lost track of this! But I know of no reason for the PR not to still be useful, so let's merge.

[ocefpaf]

Thanks @pkgw!

[mingwandroid]

but because of how libtool works, lots of libraries which link to libX11 and libxcb are also linking to libxcb-xlib

It's really not good to work around bugs in core build tools that infect other packages with problems instead of fixing the source problem. Here is the fix I wrote for libtool on defaults: https://github.com/AnacondaRecipes/libtool-feedstock/blob/master/recipe/0001-link-as-needed.patch

I will make a PR if/when conda-forge adopts our compilers and conda-build 3. It relies on passing -Wl,--as-needed as a linker flag (I have not seen any examples of where doing that causes problems, so far and if I find one, it is very easy to mask that flag off in build.sh).

[gqmelo]

@mingwandroid, this particular problem was on the libtool provided by SUSE 11, which made all system libraries which depend on libxcb also link to the private libxcb-xlib.so.0. So there's nothing else we can do except demand all SUSE 11 packages to be rebuilt (which obviously is not gonna happen).

conda-forge reached a point where we are providing some more core libraries, like libxcb, and this will naturally expose some weird bugs when interacting with other core libraries not provided by conda-forge. If the aim is to have binaries as portable and relocatable as possible, this kind of workaround is sometimes inevitable.

[mingwandroid]

@mingwandroid, this particular problem was on the libtool provided by SUSE 11, which made all system libraries which depend on libxcb also link to the private libxcb-xlib.so.0. So there's nothing else we can do except demand all SUSE 11 packages to be rebuilt (which obviously is not gonna happen).

Why can you not depend on the libtool package? (FWIW nearly every released libtool package suffer from over-linking, a few distros used to apply a fix to prevent over-linking executables but I'm not sure they've been kept up to date).

[gqmelo]

Depending on conda-forge's libtool would not help because of system libraries that cannot/should not be replaced. In this case particularly I was running against SUSE 11's libGL.so.1, which wrongly links to libxcb-xlib.so.0. I cannot rebuild libGL.so.1, this should be provided by the distro. There are lots of other libraries on SUSE 11 linking to libxcb-xlib and as can be seen on xcb mailing list this was a common problem for some older distros:

https://lists.freedesktop.org/archives/xcb/2009-April/004489.html https://lists.freedesktop.org/archives/xcb/2009-April/004492.html

I already left the job where I faced this problem, but that was the only issue blocking us from running on SUSE 11 using conda-forge packages by the time. We generated an internal libxcb package with this patch and then were able to run on SUSE 11 alongside with CentOS 6, CentOS 7, Ubuntu 14.04 and Ubuntu 16.04. But feel free to revert the patch as probably not many people try to run conda-forge's Qt on SUSE 11 and my ex-coworkers are already relying on an internal package with this change.

[mingwandroid]

conda-forge's libtool absolutely can be used to link against system provided .la files and, if you use the one in defaults it will prevent over-linking.

[gqmelo]

Sure, but in this case we were loading SUSE's libGL.so.1 at runtime, which was already linking to libxcb-xlib. Using a different libtool would not change that as libGL would not be rebuilt using the new libtool.

[mingwandroid]

Providing your own X11 and linking it to Qt is the real cause of these problems.

[gqmelo]

Sorry, I don't mean to spam all the participants, but I don't understand your point. I was not providing my own libx11 nor Qt. I even gave a reproducible example above using only conda and python (you can use SUSE 11 free trial image to try on a VM).

The public X.Org libraries are backward and forward compatible in general, so it's fine to replace libx11 and libxcb (as conda-forge does) with a newer version as long as you also provide compatible versions of the other X.Org libraries (e.g. libX11, libxcb. xinerama, xdamage, xdamage, xcursor, etc.).

But in this case the root cause is system libraries which cannot be bundled (libGL.so.1) linking to a private X.Org library (libxcb-xlib) that should only be used by libx11. Again I'm fine if anyone wants to revert this, as it's indeed a workaround. I just want to make sure that everyone understood the problem and that there's no beautiful way to make conda-forge's Qt work on SUSE 11 (which is about the same age as CentOS 6) and possibly other distros from the same time.