Latest manylinux images missing `sudo`

[tbirdso]

Summary

Local and CI tests in relation to https://github.com/InsightSoftwareConsortium/ITKPythonPackage/pull/233 fail to build on Linux. The issue appears to originate from the dockcross manylinux image.

Observed behavior

Building wheels for x64 using manylinux_2_28
++ type ninja
++ MANYLINUX_VERSION=_2_28
++ echo 'Building wheels for x64 using manylinux_2_28'
+ sudo ldconfig
/opt/rh/gcc-toolset-12/root/usr/bin/sudo: line 41: /usr/bin/sudo: No such file or directory
Error: Process completed with exit code 127.

See log: https://github.com/InsightSoftwareConsortium/ITKSplitComponents/actions/runs/3583756532/jobs/6029570633

Expected behavior

Build succeeds.

Steps to Reproduce

> MANYLINUX_VERSION=_2_28
> IMAGE_TAG=20221128-2024e4b

# Generate dockcross scripts
> docker run --rm dockcross/manylinux${MANYLINUX_VERSION}-x64:${IMAGE_TAG} > /tmp/dockcross-manylinux-x64
> chmod u+x /tmp/dockcross-manylinux-x64

# Launch container
> /tmp/dockcross-manylinux-x64 bash

# Try to run any command with sudo
>> sudo ldconfig
/opt/rh/gcc-toolset-12/root/usr/bin/sudo: line 41: /usr/bin/sudo: No such file or directory

# Try to find sudo in /usr/bin
>> ls /usr/bin | grep sudo
# no results

Other Notes

Possibly related to the most recent image tag update: https://github.com/InsightSoftwareConsortium/ITKPythonPackage/pull/230

Investigation shows that sudo works in the previous dockcross image:

> IMAGE_TAG=20221108-102ebcc
> docker run --rm dockcross/manylinux${MANYLINUX_VERSION}-x64:${IMAGE_TAG} > /tmp/dockcross-manylinux-x64
> chmod u+x /tmp/dockcross-manylinux-x64
> /tmp/dockcross-manylinux-x64 bash
>> sudo ldconfig
>> sudo echo "sudo works"
sudo works

[tbirdso]

@dzenanz Do you have bandwidth to take a look at this? I am not very familiar with the release process for dockcross images.

[dzenanz]

Even stranger is this result. Success for Python 3.7 and 3.8, failure for 3.9 and 3.10 with

+ sudo ldconfig
/opt/rh/gcc-toolset-12/root/usr/bin/sudo: line 41: /usr/bin/sudo: No such file or directory
Error: Process completed with exit code 127.

Also, MacOS packages failed due to download error. This makes me think that your initial diagnosis about missing sudo is not quite right.

[tbirdso]

Even stranger is this result. Success for Python 3.7 and 3.8, failure for 3.9 and 3.10 with

Yes, I am confused as to why 3.7 and 3.8 would succeed but 3.9 and 3.10 would fail. It is clear that sudo ldconfig succeeds in 3.7 and 3.8 and fails in 3.9 and 3.10.

3.7:

...
++ MANYLINUX_VERSION=_2_28
2022-11-29T20:58:36.1092403Z ++ echo 'Building wheels for x64 using manylinux_2_28'
2022-11-29T20:58:36.1095494Z Building wheels for x64 using manylinux_2_28
2022-11-29T20:58:36.1098387Z + sudo ldconfig
2022-11-29T20:58:36.2215625Z + export LD_LIBRARY_PATH=/opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst:/work/oneTBB-prefix/lib:/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64
...

3.10:

++ MANYLINUX_VERSION=_2_28
++ echo 'Building wheels for x64 using manylinux_2_28'
+ sudo ldconfig
/opt/rh/gcc-toolset-12/root/usr/bin/sudo: line 41: /usr/bin/sudo: No such file or directory
Error: Process completed with exit code 127.

Also, MacOS packages failed due to download error.

I believe these are two unrelated issues. Both MacOS attempts in the links you've provided failed at the CastXML download step long after the build has started, while Linux builds fail before the build begins.

From MacOS attempts:

FAILED: Wrapping/Generators/CastXML/castxml-prefix/src/castxml-stamp/castxml-download /Users/runner/work/ITKIOOMEZarrNGFF/ITKIOOMEZarrNGFF/_skbuild/macosx-10.9-universal2-3.11/cmake-build/Wrapping/Generators/CastXML/castxml-prefix/src/castxml-stamp/castxml-download 

This makes me think that your initial diagnosis about missing sudo is not quite right.

In "Other Notes" in the issue description above I was able to reproduce the issue locally by comparing sudo across the previous versus updated dockcross tag. Do you see the same results when you attempt to reproduce?

[dzenanz]

I didn't attempt to reproduce directly. Also, aren't the same docker images used for building packages for all versions of Python?

[tbirdso]

Also, aren't the same docker images used for building packages for all versions of Python?

Yes, all Linux builds for a certain specialization ("-manylinux_2_28", "-manylinux2014") are built using the same dockcross image. However, the v5.3.0 module wheels that have been built and released so far were built with the ITKPythonPackage scripts packaged with ITKPythonBuilds archives, i.e. the scripts at the ITKPythonPackage v5.3.0 tag. The updated image referenced in https://github.com/InsightSoftwareConsortium/ITKPythonPackage/pull/230 has not been successfully used to release Linux v5.3.0 wheels yet.

[thewtex]

Will be addressed via https://github.com/dockcross/dockcross/pull/747