search

ros2 / ci

ROS 2 CI Infrastructure
http://ci.ros2.org/
Apache License 2.0
48 stars 30 forks source link

Pin to RHEL 9.1 waiting pyside2 rebuild on EPEL #726

Closed claraberendsen closed 9 months ago

claraberendsen commented 9 months ago

Description

Since a couple of days ago nightlies on RHEL have been failing with: 

00:01:05.746 Error: 
00:01:05.746  Problem: package python3-pyside2-devel-1:5.15.7-2.el9.x86_64 requires libpyside2.cpython-39-x86_64-linux-gnu.so.5.15()(64bit), but none of the providers can be installed
00:01:05.746   - conflicting requests
00:01:05.746   - nothing provides libQt5Charts.so.5(Qt_5.15.3_PRIVATE_API)(64bit) needed by python3-pyside2-1:5.15.7-2.el9.x86_64

It seems that there has been a RHEL 9 release of qt 5.15.9 and that the pyside2 package has since been broken due to a dependency on the 5.15.3 private API. The release likely solves the security issue on version 5.15.3 of qt.

To resolve this issue this PR ping the version to 9.1 of the docker image that has not been updated.

claraberendsen commented 9 months ago

@clalancette from my understanding this change even though is to create_jenkins_job.py what it does is change the default args to be 9.1 instead of 9. Therefore I can test it by running CI changing those values. I believe this was put in place by Scott in #701

Almalinux 8: Build Status Almalinux 9.1:Build Status

claraberendsen commented 9 months ago

The 9.1 build seems to have failed to find libclang.so.15 it seems that in almalinux 9 is in the Devel package repository instead of AppStream ( See https://pkgs.org/search/?q=libclang.so.15).

cottsay commented 9 months ago

Is there a reason you're targeting 9.1 instead of 9.2? The current release is 9.3.

claraberendsen commented 9 months ago

Is there a reason you're targeting 9.1 instead of 9.2? The current release is 9.3.

I understood the current release was 9.2 :laughing: . I can target 9.2 to see if it produces any other issue.

cottsay commented 9 months ago

it seems that in almalinux 9 is in the Devel package repository instead of AppStream ( See https://pkgs.org/search/?q=libclang.so.15).

I think this might be an AlmaLinux-specific offering that isn't available on RHEL, so I'm not sure we should use it here. That should be confirmed, though.

claraberendsen commented 9 months ago

Almalinux 9.2 Build Status Pointing to master branch of ci/ros2

claraberendsen commented 9 months ago

I think this might be an AlmaLinux-specific offering that isn't available on RHEL, so I'm not sure we should use it here. That should be confirmed, though.

You might be right what I can see here is:

Content in the Devel repo includes packages that are not normally provided in the base nor extra repositories, but needed for build-time dependencies of other packages. Devel is NOT meant to satisfy runtime dependencies or for long term use on general purpose machines.

Here is a summary of the results so far:

@cottsay since I have you in this thread, thoughts on how to move forward? Is there somewhere else we can pull pyside2 that is not EPEL?

claraberendsen commented 9 months ago

Looking at the release date of Almalinux 9.3 of Nov 13th, this smells like changes to changes on the package upstream in preparation for the new release since the nightlies started to fail on Nov 12th. However, the old packages should still be available on 9.2, it's odd that is failing.

cottsay commented 9 months ago

thoughts on how to move forward?

This is tricky. The packaging issues we've seen in the past have been rooted in SOVERSION bumps during RHEL minor releases breaking EPEL packages, which is a pretty straightforward and predictable problem. What I think we're seeing here is a rather innocuous patch release of Qt that was pushed to the stable (9.2) release of RHEL that shouldn't have broken anything, but pyside2 has a tight coupling to the Qt version that seems to necessitate tandem builds.

Looking at the release history of pyside2 in Fedora, there are a lot of release bumps and rebuilds caused by updates to Qt, so I think this is probably a well known problem with Pyside2. Super bummer.

The only repository hand waving I can think of is to try blocking installation of the newer Qt binaries. If the older ones aren't available in the repository anymore (likely) we may need to install them from the 9.1 repositories, but get the rest of the packages from 9.2/9.3 as usual. Since the Qt releases have the same SOVERSION, it should be safe to do unless there are other packages which also depend on Qt private API.

claraberendsen commented 9 months ago

Manually linking the EPEL 8 qt5charts package seems to have resulted in another error Build Status 

Error: 
17:32:03  Problem: package python3-pyside2-devel-1:5.15.7-2.el9.x86_64 requires shiboken2, but none of the providers can be installed
17:32:03   - conflicting requests
17:32:03   - nothing provides libclang.so.15()(64bit) needed by shiboken2-1:5.15.7-2.el9.x86_64
17:32:03   - nothing provides libclang.so.15(LLVM_13)(64bit) needed by shiboken2-1:5.15.7-2.el9.x86_64

shiboken2 is also provided by EPEL tracking down provider for libclang.so.15

cottsay commented 9 months ago

I don't think you'll be able to install any RHEL 8 (or EPEL 8) packages on any version of RHEL 9.

Qt 5.15.3 is available in the RHEL 9.1 repos, but I no longer think this is a viable route forward. The EPEL update that brought newer versions of qtcharts and qtwebengine is also a problem, and that rabbit hole may go deeper. Trying to downgrade all of the qt packages across the RHEL repositories and EPEL may be unmaintainable or may not be possible at all.

The issue related to shiboken2 and clang is separate, and I think is related to the RHEL 9.3 release moving to clang 16. As it happens, shiboken2 is a subpackage of python-pyside2, so whatever solution we arrive at may mitigate both problems.

It's a mess. Part of it is EPEL's fault for not tracking RHEL minor releases very well, but this whole pyside2-private-API thing is pretty awful and unpredictable.

claraberendsen commented 9 months ago

Yes, sadly this is a hot mess. Trying to manually plumb qtcharts by itself might have worked if that was the only divergence but it's not. Most of qt dependencies have been updated on EPEL 9 to work with AlmaLinux 9.3, and since EPEL doesn't manage minor version or and archive the compatible packages for 9.2 that were working before are no longer available.

One solution in the future could be to maintain our own archive of EPEL packages to be able to revert on breaking changes and not depend on the bleeding edge EPEL major version repo. However, this means hosting them and establishing the mechanism for updating them.

I'm giving it a last try now to see if I see the end of the dependencies rabbit hole, otherwise if we want to restore service we might have to revert all the way back to RHEL 8 while we wait for pyside2 rebuild. (@clalancette)

clalancette commented 9 months ago

How about a different tactic here. Is there any chance we could take a cached Docker image off of one of the workers and pin to that Docker image? It isn't ideal, but hopefully it is a temporary measure.

claraberendsen commented 9 months ago

How about a different tactic here. Is there any chance we could take a cached Docker image off of one of the workers and pin to that Docker image? It isn't ideal, but hopefully it is a temporary measure.

I think I have found one of the agents with old docker images. Copying locally to my computer now and will check were can we put them to fetch them from. It isn't ideal but I think it's the only workaround to at least have RHEL 9.2 testing and builds. This will effectually ping us to RHEL 9.2 until we can go back to our normal workflow.

claraberendsen commented 9 months ago

Closing this in favour of #728 since strategy has changed. That PR however won't solve the underlying issue linked to this PR but should get us back working at least on ci.ros2.org.