make deepspeed fails with python-libs

[dthurston]

The install works fine until I hit the following error. I'm on RHEL 9.3.

(py-venv) [dthursto@virt rhelai-dev-preview]$ make deepspeed make[1]: Entering directory '/home/dthursto/py-venv/rhelai-dev-preview/training' make -C deepspeed/ image make[2]: Entering directory '/home/dthursto/py-venv/rhelai-dev-preview/training/deepspeed' rm -rf ../build/deepspeed-trainer "podman" build \ \ --file Containerfile \ --layers=false \ --squash-all \ --tag oci:../build/deepspeed-trainer . STEP 1/13: FROM nvcr.io/nvidia/cuda:12.1.1-cudnn8-devel-ubi9 STEP 2/13: RUN dnf install -y python python-devel git Updating Subscription Management repositories. Unable to read consumer identity subscription-manager is operating in container mode.

This system is not registered with an entitlement server. You can use subscription-manager to register.

cuda 12 MB/s | 1.5 MB 00:00
CentOS-7 82 MB/s | 11 MB 00:00
Red Hat Enterprise Linux 9 for x86_64 - BaseOS 82 MB/s | 17 MB 00:00
Red Hat Enterprise Linux 9 for x86_64 - AppStre 5.4 MB/s | 29 MB 00:05
Red Hat Universal Base Image 9 (RPMs) - BaseOS 3.0 MB/s | 515 kB 00:00
Red Hat Universal Base Image 9 (RPMs) - AppStre 12 MB/s | 2.1 MB 00:00
Red Hat Universal Base Image 9 (RPMs) - CodeRea 1.9 MB/s | 274 kB 00:00
Error: Problem 1: problem with installed package python3-librepo-1.14.5-1.el9.x86_64

• package python3-librepo-1.14.5-1.el9.x86_64 from @System requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-librepo-1.14.2-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-librepo-1.14.2-3.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-librepo-1.14.5-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-librepo-1.14.5-2.el9.x86_64 from ubi-9-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libs-3.9.18-1.el9_3.x86_64 from @System conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.10-2.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.10-3.el9_0.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9_1.2.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9_1.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9_2.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9_2.2.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-1.el9_3.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-1.el9_3.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-3.el9.x86_64 from ubi-9-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7 requires python-libs(x86-64) = 2.7.5-89.el7, but none of the providers can be installed
• package python-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7 requires libpython2.7.so.1.0()(64bit), but none of the providers can be installed
• conflicting requests Problem 2: problem with installed package python3-libdnf-0.69.0-6.el9_3.x86_64
• package python3-libdnf-0.69.0-6.el9_3.x86_64 from @System requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.65.0-5.el9_0.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.65.0-5.1.el9_0.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.67.0-3.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.69.0-3.el9_2.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.69.0-6.el9_3.x86_64 from rhel-9-for-x86_64-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libdnf-0.69.0-8.el9.x86_64 from ubi-9-baseos-rpms requires libpython3.9.so.1.0()(64bit), but none of the providers can be installed
• package python3-libs-3.9.18-1.el9_3.x86_64 from @System conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.10-2.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.10-3.el9_0.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9_1.2.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.14-1.el9_1.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9_2.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.16-1.el9_2.2.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-1.el9_3.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-1.el9_3.1.x86_64 from rhel-9-for-x86_64-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python3-libs-3.9.18-3.el9.x86_64 from ubi-9-baseos-rpms conflicts with python-libs < 3 provided by python-libs-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7
• package python-devel-2.7.5-89.el7.x86_64 from example_com_CentOS_CentOS-7 requires libpython2.7.so.1.0()(64bit), but none of the providers can be installed
• conflicting requests (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Error: building at STEP "RUN dnf install -y python python-devel git": while running runtime: exit status 1 make[2]: [Makefile:10: image] Error 1 make[2]: Leaving directory '/home/dthursto/py-venv/rhelai-dev-preview/training/deepspeed' make[1]: [Makefile:55: deepspeed] Error 2 make[1]: Leaving directory '/home/dthursto/py-venv/rhelai-dev-preview/training' make: *** [Makefile:47: deepspeed] Error 2

[ivanbaldo]

Didn't happen to me when building recently. If you can't reproduce the error anymore, can you please close the issue? Thanks!!!