search

ros2 / ci

ROS 2 CI Infrastructure
http://ci.ros2.org/
Apache License 2.0
48 stars 30 forks source link

Redo how we install pip packaging. #764

Closed clalancette closed 4 months ago

clalancette commented 4 months ago

The current way we install packages in main is really a mess. Some of the packages have pinned versions, some do not. Some of the packages are installed only if they are not installed by the underlying OS, some are not.

This commit partially cleans this up. In particular, for all pip packages that are not colcon-related, we have a tuple which contains the package name, the import name, and the (optional) version we want installed. For every pip package name, we see if we can import it (using the import name); if we can, then we don't need to install it and we skip it. If we do need to install it, then we add it to the list and add the version to the constraints.

This should mean that this code works on all of Ubuntu, RHEL, and Windows. On Ubuntu and RHEL, almost all packages are installed via the Dockerfiles (the one exception being colcon). For Windows, almost nothing is installed by the chef scripts, so most things will be installed here.

A further cleanup here would to also install colcon from packages rather than pip, but this is large enough as it is. If this is acceptable, we can do that in a followup.

Along with some other fixes, this change allows us to actually launch an Ubuntu 24.04 job (it still won't pass tests, but that is due to code-related reasons which we are fixing separately).

While this can be merged independently, in order for Ubuntu 24.04 to work we also need to merge in https://github.com/ros2/ci/pull/762 and https://github.com/ros2/ci/pull/763

clalancette commented 4 months ago

Full CI on this one:

clalancette commented 4 months ago

After the latest work which should fix the outstanding issues, here is another try at CI:

clalancette commented 4 months ago

Another try at Windows after the addition of the latest set of packages, which should make this exactly the same as the nightlies now:

clalancette commented 4 months ago

One more really quick Linux run, just to ensure that the latest change works:

clalancette commented 4 months ago

OK. There are more failures on Windows Debug than I would like, but it has also been broken for about a week so I'm not super surprised. The remaining failures mostly seem to revolve around timeouts in cpplint/cppcheck, which we can probably increase the timeouts for. None of it seems connected with the change in Python packages here. So in order to make progress on switching to Noble, I'm going to merge this one in; we'll work on improving Windows Debug in the next few weeks.

Thank you to everyone for the reviews.