Gentoo removing python2 support in some packages (breaking ROS build)

[awesomebytes]

Hello,

I've recently noticed my CI builds breaking cause some packages are being removed from the support for python2 targets.

For now I found:

• dev-util/rosdep (which we have an overlay on dev-python/rosdep instead, weird, I opened a bug on Gentoo to try to earn some time to not break my builds https://bugs.gentoo.org/708552 and try to move to py3)
• dev-python/rospkg (which we have an overlay but it's pretty old, 1.1.4 vs 1.2.3 upstream in pip)

So with this issue I would like to ask for 2 things:

1) Add overlays of what's needed to support Python 2 until the EOF of Ubuntu 18 (April 2023) which would be the EOF of Melodic I think. 2) Some help on how to emerge both py2 and py3 versions of ROS packages. I just tried and I got stuck in the... not finding gencpp, genlisp... etc cause they are generated for Py2 only, not for Py3.

I'm obviously up to report and help. But I'd like to know there will be some support.

[flabrosse]

I've spent the last few days trying to update my system, due to the mess of python2 and python3, which I made worse by trying to emerge ros. I think I now have a configuration that will emerge, fingers crossed.

However, tf is still needed by many packages in melodic, and still only works with python2. Are there plans to update this package? I don't know python much so can't really assess if going from 2 to 3 is a big job. Or is it time to go to a newer version of ROS (knowing that some of my collaborators still use kinetic!).

[allenh1]

However, tf is still needed by many packages in melodic, and still only works with python2. Are there plans to update this package? I don't know python much so can't really assess if going from 2 to 3 is a big job. Or is it time to go to a newer version of ROS (knowing that some of my collaborators still use kinetic!).

I highly encourage you to move to tf2 instead of tf if at all possible. The latest release, noetic supports python3 by default, so I'd encourage you to go there. Or, if you're feeling really brave, I'd like to get ROS 2 Foxy working on Gentoo. Though I already generate ebuilds for Foxy and the like, I have not successfully emerged on my own systems and or made the packages work.

@flabrosse I'll spend some time in the near future getting my own Gentoo system up and running with noetic, I have not tested it yet.

I'll also start working on Foxy, if I get around to it. I did get a good amount of it done, but will need to figure out the eclass some more.

[flabrosse]

Hi @allenh1, ROS 2 is not an option, as I said, some of my collaborators still use kinetic... Also, I am not the one using tf, some packages depend on it:

equery depends tf
 * These packages depend on tf:
ros-melodic/diagnostic_common_diagnostics-1.9.3 (ros-melodic/tf)
ros-melodic/geometry-1.12.1-r1 (ros-melodic/tf)
ros-melodic/hector_mapping-0.4.1-r1 (ros-melodic/tf)
ros-melodic/interactive_marker_tutorials-0.10.5-r1 (ros-melodic/tf)
ros-melodic/interactive_markers-1.11.5-r1 (ros-melodic/tf)
ros-melodic/laser_geometry-1.6.5-r1 (ros-melodic/tf)
ros-melodic/robot_state_publisher-1.14.1-r1 (ros-melodic/tf)
ros-melodic/rqt_nav_view-0.5.7 (ros-melodic/tf)
ros-melodic/rqt_pose_view-0.5.8 (ros-melodic/tf)
ros-melodic/rviz-1.13.13-r1 (ros-melodic/tf)
ros-melodic/tf_conversions-1.12.1-r1 (ros-melodic/tf)
ros-melodic/turtle_tf-0.2.2 (ros-melodic/tf)
ros-melodic/warehouse_ros-0.9.4-r1 (ros-melodic/tf)

[flabrosse]

Help! ;-)

Can't catkin_make any more. This insists on using python2, but most packages are not compiled with python2. So the build fails with not finding catkin_pkg. I guess I could force the use of the overlay version (which still supports 2_7) but I'm not convinced this will not break at some other package.

Any idea on how to force catkin_make to use python3?

[flabrosse]

Actually, I just tried to recompile the overlay catkin_pkg with python2_7 and emerge wants me to remove this version of python...

[flabrosse]

The running of python2 seems to be coming from empy, still built with python2_7 support, because tf only supports python2_7. Because of that, I can't even reemerge empy. And tf is needed by many packages (see above).

This to me indicates that melodic is dead on gentoo.

[flabrosse]

It seems that python2 is specified in

/opt/ros/melodic/etc/catkin/profile.d/1.ros_python_version.sh

created by package ros_environment. For some reason the value in there is 2, despite the package being compiled without support for python2_7.

If I change the value to 3 in this file and remove devel and build (maybe only one is needed) from my workspace, then python3 is used (even after reverting to 2 in the file above).

Seems there's a bug in ros_environment.

[flabrosse]

What I feared happened. I was in the process of upgrading a robot that had not been upgraded since March. To simplify the task I removed all the ROS packages. I now can't install ROS anymore because I can't install empy because of the reason explained a few posts above. And at the moment I don't seem to be able to install noetic (due to missing package, see #997).

[flabrosse]

More specific info on the issue:

emerge -av ros-melodic/catkin

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/empy[python_targets_python2_7(-)?,python_targets_python3_6(-)?,-python_single_target_python2_7(-),-python_single_target_python3_6(-)]".
!!! One of the following packages is required to complete your request:
- ros-melodic/catkin-0.7.26-r1::ros-overlay (Change USE: -python_targets_python2_7, this change violates use flag constraints defined by ros-melodic/catkin-0.7.26-r1: 'any-of ( python_targets_python2_7 python_targets_python3_6 )')
(dependency required by "ros-melodic/catkin-0.7.26-r1::ros-overlay" [ebuild])
(dependency required by "ros-melodic/catkin" [argument])

[flabrosse]

And the same happens with noetic.

[allenh1]

@flabrosse wow -- lots of issues there. Let me see what I can do... I'll start by updating things in the overlay, then I'll bust out my docker environment and try and get things going.

[flabrosse]

Thanks @allenh1 for looking into this. Just synced the protage tree as well as the overlay and tried to install one of the noetic packages:

emerge -av ros-noetic/costmap_2d

These are the packages that would be merged, in order:

Calculating dependencies... done!

emerge: there are no ebuilds built with USE flags to satisfy "dev-python/empy[python_targets_python2_7(-)?,python_targets_python3_6(-)?,-python_single_target_python2_7(-),-python_single_target_python3_6(-)]".
!!! One of the following packages is required to complete your request:
- ros-noetic/costmap_2d-1.17.1-r1::ros-overlay (Change USE: -python_targets_python2_7, this change violates use flag constraints defined by ros-noetic/costmap_2d-1.17.1-r1: 'any-of ( python_targets_python2_7 python_targets_python3_6 )')
(dependency required by "ros-noetic/costmap_2d-1.17.1-r1::ros-overlay" [ebuild])
(dependency required by "ros-noetic/costmap_2d" [argument])

noetic still tries to install support for python2_7 (PYTHON_COMPAT=( python{2_7,3_5,3_6} ) from the ebuild), which fails. Can't 2_7 just be removed?

[allenh1]

@flabrosse yep! I'll patch superflore accordingly (and add 3.7-3.9 while I'm at it).

[flabrosse]

Would that imply that all versions of python will need to be installed? The fact that it is trying to install for 2.7 and fails because not installed makes me think it would fail on 3.9 (still masked in the tree).

[allenh1]

Would that imply that all versions of python will need to be installed?

No, it would allow you to use any version of python 3 for noetic. Noetic does not support python 2.

[flabrosse]

ros-noetic/tf-1.13.2-r1 has python2_7 as its only python target. Yet, when trying to install it, it says:

CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find PythonInterp: Found unsuitable version "2.7.18", but
  required is at least "3" (found /usr/bin/python2.7)

So it seems to want to use python 3 but is getting python 2 because of the ebuild I guess.

[flabrosse]

Pull request #1006 to fix this submitted.

[flabrosse]

Now some of noetic do not emerge any more because they require python3_6, which seems to have been made not available. At least this is how I interpret the round brackets in:

!!! The ebuild selected to satisfy "ros-noetic/tf" has unmet requirements.
- ros-noetic/tf-1.13.2-r1::ros-overlay USE="-test" ABI_X86="(64)" PYTHON_TARGETS="(-python3_6)"

If someone knows any better, please tell me.

When I have time I'll have a go at using a more recent version of python for these packages, but I fear this will be a long and difficult task.

(I'm really considering dropping gentoo because of the management of python versions and the fact that I have no time to fight this kind of issue...)

[awesomebytes]

@flabrosse First, thanks for the continuous fight to get this working! I currently have no time to dedicate to any of this (as I am not actively using gentoo/ros-overlay) so I really appreciate it.

That said, may I ask you why/how are you using Gentoo for your project(s)? For some context, I use(d) it because using Gentoo Prefix was the only workaround to deploy modern ROS (and other state of the art SW) into unsupported platforms (and without root access). For normal deployments being able to use Ubuntu/Debian seems way more convenient, specially with the ability to put stuff into Docker containers.

[flabrosse]

Hi @awesomebytes,

I have been a gentoo user for a very long time, because I always liked the fact that you can make your install exactly what you want it to be, not what some group has decided what should be installed. As an example, we used to use an old EeePC as a convenient portable platform when following robots around, and I was maintaining gentoo on it. Despite its specs, it was perfectly usable. Then I ran out of time to maintain it and upgrading gentoo on it was taking too much time. I tried Ubuntu. That made the EeePC completely unusable. Similarly, I like my robots to spend CPU cycles on what matters, not running services that are not needed or relevant.

But the more it goes, the less time and motivation I find for upgrading OSs...

[flabrosse]

And just to check, I am bumping the package versions to r2 (or one up from whatever is there), removing the old version. In terms of python I am only keeping the versions that are still in gentoo (3_7,3_8). I noticed that 3_9 has appeared, but I am worried that the packages might not work with 3_9.

I hope this is OK.

[flabrosse]

Thinking about it, I am not sure upping the package version number is a good idea, as this might invalidate some still valid PRs still to be processed. @awesomebytes, @allenh1, what do you think?

[flabrosse]

No answer and I need to progress this.

I now have a script that will do all of this automagically so doing it again makes no difference. I'll do it without changing the release number first and redo it if numbers need to be increased.