Bug: PPC64 Docker builds are broken

[bencourliss]

Describe the bug.

The Docker builds for the ppc64 arch are failing with the following error

2023-11-14T19:59:41.0793744Z #7 ERROR: process "/bin/sh -c microdnf update  -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager   && microdnf install -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager ${REQUIRED_RPMS}   && microdnf clean all --disableplugin=subscription-manager   && rm -rf /mnt/rootfs/var/cache/* /mnt/rootfs/var/log/dnf* /mnt/rootfs/var/log/yum.*   && mkdir -p /licenses /usr/horizon/bin /usr/horizon/web /var/horizon   && mkdir -p /etc/horizon/agbot/policy.d /etc/horizon/policy.d /etc/horizon/trust /etc/docker/certs.d   && adduser agentuser -u 1000 -U -f -1 -c \"agent user,1,2,3\"" did not complete successfully: exit code: 1
2023-11-14T19:59:41.0796845Z ------
2023-11-14T19:59:41.0800484Z  > [ 3/11] RUN  microdnf update  -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager   && microdnf install -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager openssl ca-certificates shadow-utils jq iptables vim-minimal psmisc procps-ng   && microdnf clean all --disableplugin=subscription-manager   && rm -rf /mnt/rootfs/var/cache/* /mnt/rootfs/var/log/dnf* /mnt/rootfs/var/log/yum.*   && mkdir -p /licenses /usr/horizon/bin /usr/horizon/web /var/horizon   && mkdir -p /etc/horizon/agbot/policy.d /etc/horizon/policy.d /etc/horizon/trust /etc/docker/certs.d   && adduser agentuser -u 1000 -U -f -1 -c "agent user,1,2,3":
2023-11-14T19:59:41.0803366Z 1.067 Downloading metadata...
2023-11-14T19:59:41.0803682Z 3.078 Downloading metadata...
2023-11-14T19:59:41.0803991Z 6.196 Downloading metadata...
2023-11-14T19:59:41.0804303Z 7.775 Downloading metadata...
2023-11-14T19:59:41.0804751Z 30.58 error: Could not depsolve transaction; 1 problem detected:
2023-11-14T19:59:41.0805557Z 30.58  Problem: cannot install the best update candidate for package libidn2-2.3.0-7.el9.ppc64le
2023-11-14T19:59:41.0806468Z 30.58   - nothing provides libunistring.so.0()(64bit) needed by libidn2-2.3.4-1.el7.ppc64le
2023-11-14T19:59:41.0807017Z ------
2023-11-14T19:59:41.0807255Z Dockerfile.ubi.ppc64el:15
2023-11-14T19:59:41.0807576Z --------------------
2023-11-14T19:59:41.0808271Z   14 |     ARG REQUIRED_RPMS="openssl ca-certificates shadow-utils jq iptables vim-minimal psmisc procps-ng"
2023-11-14T19:59:41.0809311Z   15 | >>> RUN  microdnf update  -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager \
2023-11-14T19:59:41.0810406Z   16 | >>>   && microdnf install -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager ${REQUIRED_RPMS} \
2023-11-14T19:59:41.0811301Z   17 | >>>   && microdnf clean all --disableplugin=subscription-manager \
2023-11-14T19:59:41.0812139Z   18 | >>>   && rm -rf /mnt/rootfs/var/cache/* /mnt/rootfs/var/log/dnf* /mnt/rootfs/var/log/yum.* \
2023-11-14T19:59:41.0812950Z   19 | >>>   && mkdir -p /licenses /usr/horizon/bin /usr/horizon/web /var/horizon \
2023-11-14T19:59:41.0813870Z   20 | >>>   && mkdir -p /etc/horizon/agbot/policy.d /etc/horizon/policy.d /etc/horizon/trust /etc/docker/certs.d \
2023-11-14T19:59:41.0814686Z   21 | >>>   && adduser agentuser -u 1000 -U -f -1 -c "agent user,1,2,3"
2023-11-14T19:59:41.0815282Z   22 |     
2023-11-14T19:59:41.0815536Z --------------------
2023-11-14T19:59:41.0820693Z ERROR: failed to solve: process "/bin/sh -c microdnf update  -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager   && microdnf install -y --nodocs --setopt=install_weak_deps=0 --disableplugin=subscription-manager ${REQUIRED_RPMS}   && microdnf clean all --disableplugin=subscription-manager   && rm -rf /mnt/rootfs/var/cache/* /mnt/rootfs/var/log/dnf* /mnt/rootfs/var/log/yum.*   && mkdir -p /licenses /usr/horizon/bin /usr/horizon/web /var/horizon   && mkdir -p /etc/horizon/agbot/policy.d /etc/horizon/policy.d /etc/horizon/trust /etc/docker/certs.d   && adduser agentuser -u 1000 -U -f -1 -c \"agent user,1,2,3\"" did not complete successfully: exit code: 1
2023-11-14T19:59:41.0842110Z make: *** [Makefile:435: anax-k8s-image] Error 1

Describe the steps to reproduce the behavior.

No response

Expected behavior.

No response

Screenshots.

No response

Operating Environment

ppc64el

Additional Information

No response

[dlarson04]

I have experienced this before where dnf stopped working... and had to get a new RHEL VM and it worked

[bencourliss]

@dlarson04 in this case it's happening in the GitHub Action container the build is running in to create the varioys Anax Docker container. Apparently adding --nobest to the microdnf command helps. https://access.redhat.com/discussions/4728791.

Also noticed that the Docker build command failed but did not fail the GitHub Action workflow. This caused older artifacts to be used downstream. The GitHub Action calls an external Bash script which then calls a Makefile. Non-zero exit codes are apparently not bubbled up to fail the GHA. In testing I added set -eo pipefail to the Bash scripts to no avail. Only thing that seemed to work was moving the commands in the Bash script into the GHA workflow yaml itself.