ploigos-tool-autogov_ubi8 failing nightly build

[jimmyliang-gov]

The ploigos-tool-autogov_ubi8 and _ubi9 image failed the publish #1240 run.

Build Issue

 > [3/5] RUN INSTALL_PKGS="golang" &&     dnf update -y --allowerasing --nobest &&     dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS &&     dnf clean all &&     rm -rf /var/cache /var/log/dnf* /var/log/yum.*:
1.000 Updating Subscription Management repositories.
1.001 Unable to read consumer identity
1.004 
1.004 This system is not registered with an entitlement server. You can use subscription-manager to register.
1.004 
1.326 centos                                           28 kB/s | 7.9 kB     00:00    
1.326 Failed to set locale, defaulting to C.UTF-8
1.326 Errors during downloading metadata for repository 'centos':
1.326   - Status code: 404 for http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/repodata/repomd.xml (IP: 18.219.147.23)
1.335 Error: Failed to download metadata for repo 'centos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
------
Containerfile.ubi8:34
--------------------
  33 |     # update and install packages
  34 | >>> RUN INSTALL_PKGS="golang" && \
  35 | >>>     dnf update -y --allowerasing --nobest && \
  36 | >>>     dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS && \
  37 | >>>     dnf clean all && \
  38 | >>>     rm -rf /var/cache /var/log/dnf* /var/log/yum.*
  39 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c INSTALL_PKGS=\"golang\" &&     dnf update -y --allowerasing --nobest &&     dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS &&     dnf clean all &&     rm -rf /var/cache /var/log/dnf* /var/log/yum.*" did not complete successfully: exit code: 1
Error: buildx failed with: ERROR: failed to solve: process "/bin/sh -c INSTALL_PKGS=\"golang\" &&     dnf update -y --allowerasing --nobest &&     dnf install -y --setopt=tsflags=nodocs $INSTALL_PKGS &&     dnf clean all &&     rm -rf /var/cache /var/log/dnf* /var/log/yum.*" did not complete successfully: exit code: 1

The CentOS mirror is no longer reachable.

Potential Solution

Use RH entitled builds (might require build server to be the subscribed with RH)

[jimmyliang-gov]

I have a version of this working without the CentOS repo and using RH repos (but it requires entitled builds)

[dwinchell]

Unfortunately, the build environment right now is the Ubuntu VMs that the GitHub Actions pipeline itself is able to stand up - https://github.com/ploigos/ploigos-containers/blob/537762d7823987a84166489b83686a3f3e4f653f/.github/workflows/publish.yaml#L32

So the easy solution of just running it on an entitled RHEL VM is not in place.

There used to be some additional infrastructure that was used to build and maintain the ploigos project including a RHEL VM that we might have used, but that has long since been de-comissioned.

Some other options I can think of:

1. Someone finds funding for, stands up, and maintains a RHEL VM, and then integrates it with the nightly build.
2. Deprecate ploigos-tool-autogov_* and remove them from the build. This is not a good option because other parts of the ploigos ecosystem rely on them, specifically the "everything" pipeline.
3. I found a blog post that says it's possible to do entitled UBI builds inside of a container on a non-RHEL host, as of 2020/UBI8. I don't know if that's still possible, or if it would be possible for UBI9.

Option 3 seems best from a technical perspective, if it works, but to be candid I do not personally have the bandwidth to devote any significant work to Ploigos at this time, which would include implementing that.

@jimmyliang-gov I wish I could be of more help. I think the root cause you're really running into here is a dependency on an older opensource project without much of a community left behind it.

[jimmyliang-gov]

The ploigos-tool-autogov image is failing on golang install (previously it was trying to install from a centos repo which has been deprecated). If entitled builds are not an option (yet) then I have another idea for installing GoLang. PR coming soon.

[jimmyliang-gov]

PR: https://github.com/ploigos/ploigos-containers/pull/125