search

IBM / cloudpak-gitops

Deployment of IBM Cloud Paks using ArgoCD / Red Hat GitOps operator.
Apache License 2.0
22 stars 23 forks source link

Update Cloud Pak for Data to release 4.5.3 #173

Closed nastacio closed 1 year ago

nastacio commented 2 years ago

Is your feature request related to a problem? Please describe. Cloud Pak for Data 4.5 has been out for a while, so we need to update this repository.

Describe the solution you'd like Update the CRs and jobs to reflect the updated installation procedures.

Describe alternatives you've considered Still considering: The installation procedure has shifted towards a completely procedural approach (using the newly introduced cpd-cli,) so it is possible we may need to drop Cloud Pak for Data support from this repository altogether.

Additional context Add any other context or screenshots about the feature request here.

nastacio commented 2 years ago
  1. The image we currently use to talk to OpenShift (quay.io/openshift/origin-cli) does not contain podman, which I tried to workaround with installing podman. That approach fails because there is no package manager installed in that image. Subsequent attempts with dnf failed with the exact same messages:
yum install podman
Updating Subscription Management repositories.
Unable to read consumer identity

This system is not registered with an entitlement server. You can use subscription-manager to register.

rhel-8-baseos                                                                                                                                                                                                 0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-baseos':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-baseos/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-baseos': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-appstream                                                                                                                                                                                              0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-appstream':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-appstream/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-appstream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-fast-datapath                                                                                                                                                                                          0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-fast-datapath':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-fast-datapath/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-fast-datapath': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-nfv                                                                                                                                                                                                    0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-nfv':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-nfv/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-nfv': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-server-ose                                                                                                                                                                                             0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-server-ose':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-server-ose/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-server-ose': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-ansible-2.9                                                                                                                                                                                            0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-ansible-2.9':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-ansible-2.9/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-ansible-2.9': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
openstack-16-for-rhel-8-rpms                                                                                                                                                                                  0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'openstack-16-for-rhel-8-rpms':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/openstack-16-for-rhel-8-rpms/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'openstack-16-for-rhel-8-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
rhel-8-codeready-builder-rpms                                                                                                                                                                                 0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'rhel-8-codeready-builder-rpms':
  - Curl error (6): Couldn't resolve host name for http://base-4-12-rhel8.ocp.svc/rhel-8-codeready-builder-rpms/repodata/repomd.xml [Could not resolve host: base-4-12-rhel8.ocp.svc]
Error: Failed to download metadata for repo 'rhel-8-codeready-builder-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-openstack-16-for-rhel-8-rpms                                                                                                                                                                         0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-openstack-16-for-rhel-8-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/layered/rhel8/x86_64/openstack/16.2/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-openstack-16-for-rhel-8-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-appstream-rpms                                                                                                                                                                                0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-appstream-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-appstream-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-baseos-rpms                                                                                                                                                                                   0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-baseos-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-baseos-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-codeready-builder-rpms                                                                                                                                                                        0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-codeready-builder-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/rhel8/8/x86_64/codeready-builder/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-codeready-builder-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-fast-datapath-rpms                                                                                                                                                                            0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-fast-datapath-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/layered/rhel8/x86_64/fast-datapath/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-fast-datapath-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-rt-rpms                                                                                                                                                                                       0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-rt-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/rhel8/8/x86_64/rt/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-rt-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-server-ansible-2.9-rpms                                                                                                                                                                       0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-server-ansible-2.9-rpms':
  - Curl error (6): Couldn't resolve host name for http://rhsm-pulp.corp.redhat.com/content/dist/layered/rhel8/x86_64/ansible/2.9/os/repodata/repomd.xml [Could not resolve host: rhsm-pulp.corp.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-server-ansible-2.9-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
localdev-rhel-8-server-ose-rpms                                                                                                                                                                               0.0  B/s |   0  B     00:00    
Errors during downloading metadata for repository 'localdev-rhel-8-server-ose-rpms':
  - Curl error (6): Couldn't resolve host name for http://download.lab.bos.redhat.com/rcm-guest/puddles/RHAOS/plashets/4.12-el8/building/x86_64/os/repodata/repomd.xml [Could not resolve host: download.lab.bos.redhat.com]
Error: Failed to download metadata for repo 'localdev-rhel-8-server-ose-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
Ignoring repositories: rhel-8-baseos, rhel-8-appstream, rhel-8-fast-datapath, rhel-8-nfv, rhel-8-server-ose, rhel-8-ansible-2.9, openstack-16-for-rhel-8-rpms, rhel-8-codeready-builder-rpms, localdev-openstack-16-for-rhel-8-rpms, localdev-rhel-8-appstream-rpms, localdev-rhel-8-baseos-rpms, localdev-rhel-8-codeready-builder-rpms, localdev-rhel-8-fast-datapath-rpms, localdev-rhel-8-rt-rpms, localdev-rhel-8-server-ansible-2.9-rpms, localdev-rhel-8-server-ose-rpms
No match for argument: podman
Error: Unable to find a match: podman
nastacio commented 2 years ago
  1. The first obvious workaround is to use image registry.redhat.io/rhel8/podman instead. The registry requires a pull secret, but that pull secret is already required for installing OCP and this repo is already setting it up in the cluster's global pull secret, so that is not a problem.

But then the first invocation of cpd-cli immediately runs into this issue:

cpd-cli manage login-to-ocp
[INFO] 2022-10-26T18:41:47.056026Z Checking architecture: amd64
[INFO] 2022-10-26T18:41:47.059507Z Checking podman or docker
[INFO] 2022-10-26T18:41:47.197908Z Dockerexe: podman
[INFO] 2022-10-26T18:41:47.199883Z Checking container image
[INFO] 2022-10-26T18:41:47.202608Z Creating directory /cpd-cli-workspace/olm-utils-workspace/work
[INFO] 2022-10-26T18:41:47.835399Z Pulling the container image icr.io/cpopen/cpd/olm-utils:latest
[INFO] 2022-10-26T18:42:16.978785Z Pulling image output: Trying to pull icr.io/cpopen/cpd/olm-utils:latest...
Getting image source signatures
...
Copying blob ...
Error: writing blob: adding layer with blob "sha256:d921a34d525b53cffec3ed5317dab0511927c3e01de4d783cdcf392e332a436b": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:12 for /var/spool/mail): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /var/spool/mail: invalid argument
[ERROR] 2022-10-26T18:42:17.001914Z Command exception: exit status 125
[ERROR] 2022-10-26T18:42:17.014112Z RunPluginCommand:Execution error:  exit status 1
nastacio commented 2 years ago
  1. The workaround for that problem is simple:
sed -i 's|.*ignore_chown_errors.*=.*"false"|ignore_chown_errors = "true"|g' /etc/containers/storage.conf

The cpd-cli went further, but then I ran into what seems the ultimate blocker:

cpd-cli manage --username=${OCP_USERNAME} --password=${OCP_PASSWORD} --server=${OCP_URL} 
[INFO] 2022-10-26T19:02:19.373358Z Checking architecture: amd64
[INFO] 2022-10-26T19:02:19.376611Z Checking podman or docker
[INFO] 2022-10-26T19:02:19.424024Z Dockerexe: podman
[INFO] 2022-10-26T19:02:19.426878Z Checking container image
[INFO] 2022-10-26T19:02:19.551983Z Pulling the container image icr.io/cpopen/cpd/olm-utils:latest
[INFO] 2022-10-26T19:02:20.495133Z Pulling image output: Trying to pull icr.io/cpopen/cpd/olm-utils:latest...
Getting image source signatures
...
Writing manifest to image destination
Storing signatures
792c15a6d4e50a42239452fe122c7b37f4c3ddca5aa4e2027fd944d75c0f711c
[SUCCESS] 2022-10-26T19:02:20.522470Z Successfully pulled the container image icr.io/cpopen/cpd/olm-utils:latest
[INFO] 2022-10-26T19:02:20.522567Z Delete any existing olm-utils-play container
[INFO] 2022-10-26T19:02:20.646390Z Error: no container with name or ID "olm-utils-play" found: no such container
[INFO] 2022-10-26T19:02:20.649088Z Checking SELinux configuration
[INFO] 2022-10-26T19:02:20.651150Z Start the olm-utils-play container
[ERROR] 2022-10-26T19:02:20.809785Z Command exception: Failed to start the olm-utils-play container: exit status 127
[ERROR] 2022-10-26T19:02:20.814819Z RunPluginCommand:Execution error:  exit status 1

The weird think is that olm-utils-play was started at that point, so it looks like the cpd-cli cannot detect that the container is running:

podman ps -a
CONTAINER ID  IMAGE                               COMMAND     CREATED        STATUS      PORTS       NAMES
e1d749a524ca  icr.io/cpopen/cpd/olm-utils:latest              7 minutes ago  Created                 olm-utils-play
nastacio commented 1 year ago

My initial assessment is that it is not possible to use cpd-cli from inside a container, but it is possible to create a proof of concept using the container loaded by cpd-cli (icr.io/cpopen/cpd/olm-utils:latest) with some reservations:

  1. It is not a documented interface, so there is no guarantee it will remain working in future releases.
  2. Using the icr.io/cpopen/cpd/olm-utils:latest container directly inside a k8s container creates additional output that confuses the bin/setup-route command and possibly others.

For now, I merged the PR that allows the installation on a green cluster, but I want to revisit this topic in the upcoming 4.6 release of CP4D.