search

crc-org / snc

Single Node Cluster creation scripts for OpenShift 4.x as used by CodeReady Containers
https://crc.dev
Apache License 2.0
100 stars 50 forks source link

Facing issues to create cluster on IBM power 9 server #736

Closed ghatwala closed 1 year ago

ghatwala commented 1 year ago

Hi all,

Using this trying to install SNC and facing below issues on a power 9 server. Attaching installation logs for reference with ./snc.sh &> snc-cmdrunlog.txt. snc-cmdrunlog.txt . Any pointers to resolve the issue ?

# ./openshift-clients/linux/oc get co
NAME                                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.0-ec.2   True        False         False      3d
config-operator                            4.13.0-ec.2   True        False         False      4d
console                                    4.13.0-ec.2   True        False         False      3d
control-plane-machine-set                  4.13.0-ec.2   True        False         False      4d
dns                                        4.13.0-ec.2   True        False         False      3d
etcd                                       4.13.0-ec.2   True        False         False      4d
image-registry                             4.13.0-ec.2   True        True          True       4d      Degraded: Registry deployment has timed out progressing: ReplicaSet "image-registry-76db78f4d5" has timed out progressing.
ingress                                    4.13.0-ec.2   True        False         False      4d
kube-apiserver                             4.13.0-ec.2   True        False         False      4d
kube-controller-manager                    4.13.0-ec.2   True        False         False      4d
kube-scheduler                             4.13.0-ec.2   True        False         False      4d
kube-storage-version-migrator              4.13.0-ec.2   True        False         False      4d
machine-api                                4.13.0-ec.2   True        False         False      4d
machine-approver                           4.13.0-ec.2   True        False         False      4d
machine-config                             4.13.0-ec.2   True        False         False      4d
marketplace                                4.13.0-ec.2   True        False         False      4d
network                                    4.13.0-ec.2   True        False         False      4d
node-tuning                                4.13.0-ec.2   True        False         False      4d
openshift-apiserver                        4.13.0-ec.2   True        False         False      3d
openshift-controller-manager               4.13.0-ec.2   True        False         False      3d
openshift-samples                          4.13.0-ec.2   True        False         False      4d
operator-lifecycle-manager                 4.13.0-ec.2   True        False         False      4d
operator-lifecycle-manager-catalog         4.13.0-ec.2   True        False         False      4d
operator-lifecycle-manager-packageserver   4.13.0-ec.2   True        False         False      3d
service-ca                                 4.13.0-ec.2   True        False         False      4d

# ./openshift-clients/linux/oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-ec.2   True        False         4d      Error while reconciling 4.13.0-ec.2: the cluster operator image-registry is degraded

Checking for logs..

# ./openshift-clients/linux/oc describe co image-registry
Name:         image-registry
Namespace:
Labels:       <none>
Annotations:  include.release.openshift.io/ibm-cloud-managed: true
              include.release.openshift.io/self-managed-high-availability: true
              include.release.openshift.io/single-node-developer: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2023-06-01T12:48:21Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:include.release.openshift.io/ibm-cloud-managed:
          f:include.release.openshift.io/self-managed-high-availability:
          f:include.release.openshift.io/single-node-developer:
        f:ownerReferences:
          .:
          k:{"uid":"29eb2a2b-d4b6-4ba2-9ee7-ef8d16f4f310"}:
      f:spec:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2023-06-01T12:48:21Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:extension:
    Manager:      cluster-version-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-06-01T12:48:21Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:relatedObjects:
        f:versions:
    Manager:      cluster-image-registry-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-06-02T14:25:47Z
  Owner References:
    API Version:     config.openshift.io/v1
    Kind:            ClusterVersion
    Name:            version
    UID:             29eb2a2b-d4b6-4ba2-9ee7-ef8d16f4f310
  Resource Version:  36615
  UID:               bf3d5457-50b6-4353-bc47-e8da02b78168
Spec:
Status:
  Conditions:
    Last Transition Time:  2023-06-01T13:04:35Z
    Message:               Available: The registry has minimum availability
NodeCADaemonAvailable: The daemon set node-ca has available replicas
ImagePrunerAvailable: Pruner CronJob has been created
    Reason:                MinimumAvailability
    Status:                True
    Type:                  Available
    Last Transition Time:  2023-06-02T13:30:34Z
    Message:               Progressing: The deployment has not completed
NodeCADaemonProgressing: The daemon set node-ca is deployed
    Reason:                DeploymentNotCompleted
    Status:                True
    Type:                  Progressing
    Last Transition Time:  2023-06-02T14:25:47Z
    Message:               Degraded: Registry deployment has timed out progressing: ReplicaSet "image-registry-76db78f4d5" has timed out progressing.
    Reason:                ProgressDeadlineExceeded
    Status:                True
    Type:                  Degraded
  Extension:               <nil>
  Related Objects:
    Group:     imageregistry.operator.openshift.io
    Name:      cluster
    Resource:  configs
    Group:     imageregistry.operator.openshift.io
    Name:      cluster
    Resource:  imagepruners
    Group:     rbac.authorization.k8s.io
    Name:      system:registry
    Resource:  clusterroles
    Group:     rbac.authorization.k8s.io
    Name:      registry-registry-role
    Resource:  clusterrolebindings
    Group:     rbac.authorization.k8s.io
    Name:      openshift-image-registry-pruner
    Resource:  clusterrolebindings
    Group:
    Name:      openshift-image-registry
    Resource:  namespaces
  Versions:
    Name:     operator
    Version:  4.13.0-ec.2
Events:       <none>

# ./openshift-clients/linux/oc get po -n openshift-image-registry
NAME                                              READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-df49cfd7f-mwpkt   1/1     Running     2          4d1h
image-pruner-28095840-wg4l2                       0/1     Completed   0          2d13h
image-pruner-28097280-qjszz                       0/1     Completed   0          37h
image-pruner-28098720-k8mwz                       0/1     Completed   0          13h
image-registry-76db78f4d5-xvp8n                   0/1     Pending     0          2d23h
image-registry-d7f77b585-mghtw                    1/1     Running     2          4d
node-ca-kcvhs                                     1/1     Running     2          4d

checking failing pod logs

# ./openshift-clients/linux/oc describe po -n openshift-image-registry image-registry-76db78f4d5-xvp8n
Name:                 image-registry-76db78f4d5-xvp8n
Namespace:            openshift-image-registry
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      registry
Node:                 <none>
Labels:               docker-registry=default
                      pod-template-hash=76db78f4d5
Annotations:          imageregistry.operator.openshift.io/dependencies-checksum: sha256:5851fee4a9fb3ce4403dbd14db0c8993ad35242c92fdd0a79f3f09722d545638
                      openshift.io/scc: restricted-v2
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/image-registry-76db78f4d5
Containers:
  registry:
    Image:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7e01095809c29cbd68b1c3c3c4a5c0b249e4fcb9d642a80f9405a208f0ee257
    Port:       5000/TCP
    Host Port:  0/TCP
    Command:
      /bin/sh
      -c
      mkdir -p /etc/pki/ca-trust/extracted/edk2 /etc/pki/ca-trust/extracted/java /etc/pki/ca-trust/extracted/openssl /etc/pki/ca-trust/extracted/pem && update-ca-trust extract && exec /usr/bin/dockerregistry
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get https://:5000/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get https://:5000/healthz delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_STORAGE:                           filesystem
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /registry
      REGISTRY_HTTP_ADDR:                         :5000
      REGISTRY_HTTP_NET:                          tcp
      REGISTRY_HTTP_SECRET:                       a16ee665091727a6e4fa49d9ac55cf62f56683c17c806380e9ca8204f00cc57878861f3e149aab984ec6cce4198427ab0ff8fd4f1e56aa01f90ea9806d8c1950
      REGISTRY_LOG_LEVEL:                         info
      REGISTRY_OPENSHIFT_QUOTA_ENABLED:           true
      REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR:      inmemory
      REGISTRY_STORAGE_DELETE_ENABLED:            true
      REGISTRY_HEALTH_STORAGEDRIVER_ENABLED:      true
      REGISTRY_HEALTH_STORAGEDRIVER_INTERVAL:     10s
      REGISTRY_HEALTH_STORAGEDRIVER_THRESHOLD:    1
      REGISTRY_OPENSHIFT_METRICS_ENABLED:         true
      REGISTRY_OPENSHIFT_SERVER_ADDR:             image-registry.openshift-image-registry.svc:5000
      REGISTRY_HTTP_TLS_CERTIFICATE:              /etc/secrets/tls.crt
      REGISTRY_HTTP_TLS_KEY:                      /etc/secrets/tls.key
    Mounts:
      /etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
      /etc/pki/ca-trust/source/anchors from registry-certificates (rw)
      /etc/secrets from registry-tls (rw)
      /registry from registry-storage (rw)
      /usr/share/pki/ca-trust-source from trusted-ca (rw)
      /var/lib/kubelet/ from installation-pull-secrets (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgps4 (ro)
      /var/run/secrets/openshift/serviceaccount from bound-sa-token (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  registry-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  crc-image-registry-storage
    ReadOnly:   false
  registry-tls:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          image-registry-tls
    SecretOptionalName:  <nil>
  ca-trust-extracted:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  registry-certificates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      image-registry-certificates
    Optional:  false
  trusted-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca
    Optional:  true
  installation-pull-secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  installation-pull-secrets
    Optional:    true
  bound-sa-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3600
  kube-api-access-hgps4:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
    ConfigMapName:            openshift-service-ca.crt
    ConfigMapOptional:        <nil>
QoS Class:                    Burstable
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
                              node-role.kubernetes.io/worker:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
Events:
  Type     Reason            Age                      From               Message
  ----     ------            ----                     ----               -------
  Warning  FailedScheduling  4m25s (x421 over 2d23h)  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition

Another pod which is failing due to missing power arch support for the image.

# ./openshift-clients/linux/oc describe po -n hostpath-provisioner csi-hostpathplugin-s5z6q
Name:             csi-hostpathplugin-s5z6q
Namespace:        hostpath-provisioner
Priority:         0
Service Account:  csi-hostpath-provisioner-sa
Node:             crc-ks9kv-master-0/192.168.126.11
Start Time:       Fri, 02 Jun 2023 09:30:28 -0400
Labels:           app.kubernetes.io/component=plugin
                  app.kubernetes.io/instance=hostpath.csi.kubevirt.io
                  app.kubernetes.io/name=csi-hostpathplugin
                  app.kubernetes.io/part-of=csi-driver-host-path
                  controller-revision-hash=687947cb65
                  pod-template-generation=1
Annotations:      k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "openshift-sdn",
                        "interface": "eth0",
                        "ips": [
                            "10.217.0.19"
                        ],
                        "default": true,
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks-status:
                    [{
                        "name": "openshift-sdn",
                        "interface": "eth0",
                        "ips": [
                            "10.217.0.19"
                        ],
                        "default": true,
                        "dns": {}
                    }]
                  openshift.io/scc: hostpath-provisioner
Status:           Pending
IP:               10.217.0.19
IPs:
  IP:           10.217.0.19
Controlled By:  DaemonSet/csi-hostpathplugin
Containers:
  hostpath-provisioner:
    Container ID:
    Image:         quay.io/crcont/hostpath-csi-driver:v4.11.0
    Image ID:
    Port:          9898/TCP
    Host Port:     0/TCP
    Args:
      --drivername=kubevirt.io.hostpath-provisioner
      --v=3
      --datadir=[{"name":"local","path":"/csi-data-dir"}]
      --endpoint=$(CSI_ENDPOINT)
      --nodeid=$(NODE_NAME)
      --version=$(VERSION)
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:9898/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
    Environment:
      CSI_ENDPOINT:  unix:///csi/csi.sock
      NODE_NAME:      (v1:spec.nodeName)
      PV_DIR:        /var/hpvolumes
      VERSION:       latest
    Mounts:
      /csi from socket-dir (rw)
      /csi-data-dir from csi-data-dir (rw)
      /var/lib/kubelet/plugins from plugins-dir (rw)
      /var/lib/kubelet/pods from mountpoint-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mbjp (ro)
  node-driver-registrar:
    Container ID:  cri-o://174ec7774d3eb99d7210c5fe47de90fa26347ec6e1a0dced4c95789196c83163
    Image:         registry.redhat.io/openshift4/ose-csi-node-driver-registrar:latest
    Image ID:      registry.redhat.io/openshift4/ose-csi-node-driver-registrar@sha256:71d42a2b26fca9690036c95e54aa3641702c4dc7ba1912f8804d777320f9fbf8
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=3
      --csi-address=/csi/csi.sock
      --kubelet-registration-path=/var/lib/kubelet/plugins/csi-hostpath/csi.sock
    State:          Running
      Started:      Fri, 02 Jun 2023 09:34:29 -0400
    Ready:          True
    Restart Count:  1
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /csi-data-dir from csi-data-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mbjp (ro)
  liveness-probe:
    Container ID:  cri-o://138fa59be3d554c781f9be8013da354355dd314b622e7cb5a0d4af9b8a7f459f
    Image:         registry.redhat.io/openshift4/ose-csi-livenessprobe:latest
    Image ID:      registry.redhat.io/openshift4/ose-csi-livenessprobe@sha256:3c63c06561f20f5ead423946c6cc22030c397f0fe190004e6b5ccdb3bfaccdc4
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=/csi/csi.sock
      --health-port=9898
    State:          Running
      Started:      Fri, 02 Jun 2023 09:34:36 -0400
    Ready:          True
    Restart Count:  1
    Environment:    <none>
    Mounts:
      /csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mbjp (ro)
  csi-provisioner:
    Container ID:  cri-o://525f8278973e63ebca2f06624ba9010b0de7d4e78edee0a2eb2b90bc443b7ba0
    Image:         registry.redhat.io/openshift4/ose-csi-external-provisioner:latest
    Image ID:      registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:72fa8d74857ed9601c8431ceccb9ce12fbafba30a40b29f47b6c1c9a69ea43f3
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=/csi/csi.sock
      --feature-gates=Topology=true
      --enable-capacity=true
      --capacity-for-immediate-binding=true
      --extra-create-metadata=true
      --immediate-topology=false
      --strict-topology=true
      --node-deployment=true
    State:          Running
      Started:      Fri, 02 Jun 2023 09:34:43 -0400
    Ready:          True
    Restart Count:  1
    Environment:
      NAMESPACE:  hostpath-provisioner (v1:metadata.namespace)
      POD_NAME:   csi-hostpathplugin-s5z6q (v1:metadata.name)
      NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5mbjp (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/csi-hostpath
    HostPathType:  DirectoryOrCreate
  mountpoint-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry
    HostPathType:  Directory
  plugins-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins
    HostPathType:  Directory
  csi-data-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/csi-hostpath-data/
    HostPathType:  DirectoryOrCreate
  kube-api-access-5mbjp:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason   Age                   From     Message
  ----    ------   ----                  ----     -------
  Normal  Pulling  26m (x848 over 3d)    kubelet  Pulling image "quay.io/crcont/hostpath-csi-driver:v4.11.0"
  Normal  BackOff  69s (x19151 over 3d)  kubelet  Back-off pulling image "quay.io/crcont/hostpath-csi-driver:v4.11.0"
praveenkumar commented 1 year ago

@ghatwala Problem with hostpath-csi-driver because the image we have is only for amd64 and arm64 arch not for power pc arch :( other than that I don't see any issue and if you don't need CSI driver, this can be ignored.

ghatwala commented 1 year ago

thanks @praveenkumar for your response and ack on ignoring the hostpath-csi-driver based pod , however in my logs above there's a cluster operator ( image-registry) which is failing/degraded state ( pod is in pending state) , capturing info again , any inputs/pointers on how to get this CO in true state.

image-registry                             4.13.0-ec.2   True        True          True       4d      Degraded: Registry deployment has timed out progressing: ReplicaSet "image-registry-76db78f4d5" has timed out progressing.
# ./openshift-clients/linux/oc get po -n openshift-image-registry
NAME                                              READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-df49cfd7f-mwpkt   1/1     Running     2          4d1h
image-pruner-28095840-wg4l2                       0/1     Completed   0          2d13h
image-pruner-28097280-qjszz                       0/1     Completed   0          37h
image-pruner-28098720-k8mwz                       0/1     Completed   0          13h
image-registry-76db78f4d5-xvp8n                   0/1     Pending     0          2d23h
image-registry-d7f77b585-mghtw                    1/1     Running     2          4d
node-ca-kcvhs                                     1/1     Running     2          4d

checking failing pod logs

# ./openshift-clients/linux/oc describe po -n openshift-image-registry image-registry-76db78f4d5-xvp8n
Name:                 image-registry-76db78f4d5-xvp8n
Namespace:            openshift-image-registry
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      registry
Node:                 <none>
Labels:               docker-registry=default
                      pod-template-hash=76db78f4d5
Annotations:          imageregistry.operator.openshift.io/dependencies-checksum: sha256:5851fee4a9fb3ce4403dbd14db0c8993ad35242c92fdd0a79f3f09722d545638
                      openshift.io/scc: restricted-v2
                      seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:               Pending
IP:
IPs:                  <none>
Controlled By:        ReplicaSet/image-registry-76db78f4d5
Containers:
  registry:
    Image:      quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7e01095809c29cbd68b1c3c3c4a5c0b249e4fcb9d642a80f9405a208f0ee257
    Port:       5000/TCP
    Host Port:  0/TCP
    Command:
      /bin/sh
      -c
      mkdir -p /etc/pki/ca-trust/extracted/edk2 /etc/pki/ca-trust/extracted/java /etc/pki/ca-trust/extracted/openssl /etc/pki/ca-trust/extracted/pem && update-ca-trust extract && exec /usr/bin/dockerregistry
    Requests:
      cpu:      100m
      memory:   256Mi
    Liveness:   http-get https://:5000/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
    Readiness:  http-get https://:5000/healthz delay=15s timeout=5s period=10s #success=1 #failure=3
    Environment:
      REGISTRY_STORAGE:                           filesystem
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY:  /registry
      REGISTRY_HTTP_ADDR:                         :5000
      REGISTRY_HTTP_NET:                          tcp
      REGISTRY_HTTP_SECRET:                       a16ee665091727a6e4fa49d9ac55cf62f56683c17c806380e9ca8204f00cc57878861f3e149aab984ec6cce4198427ab0ff8fd4f1e56aa01f90ea9806d8c1950
      REGISTRY_LOG_LEVEL:                         info
      REGISTRY_OPENSHIFT_QUOTA_ENABLED:           true
      REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR:      inmemory
      REGISTRY_STORAGE_DELETE_ENABLED:            true
      REGISTRY_HEALTH_STORAGEDRIVER_ENABLED:      true
      REGISTRY_HEALTH_STORAGEDRIVER_INTERVAL:     10s
      REGISTRY_HEALTH_STORAGEDRIVER_THRESHOLD:    1
      REGISTRY_OPENSHIFT_METRICS_ENABLED:         true
      REGISTRY_OPENSHIFT_SERVER_ADDR:             image-registry.openshift-image-registry.svc:5000
      REGISTRY_HTTP_TLS_CERTIFICATE:              /etc/secrets/tls.crt
      REGISTRY_HTTP_TLS_KEY:                      /etc/secrets/tls.key
    Mounts:
      /etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
      /etc/pki/ca-trust/source/anchors from registry-certificates (rw)
      /etc/secrets from registry-tls (rw)
      /registry from registry-storage (rw)
      /usr/share/pki/ca-trust-source from trusted-ca (rw)
      /var/lib/kubelet/ from installation-pull-secrets (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgps4 (ro)
      /var/run/secrets/openshift/serviceaccount from bound-sa-token (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  registry-storage:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  crc-image-registry-storage
    ReadOnly:   false
  registry-tls:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          image-registry-tls
    SecretOptionalName:  <nil>
  ca-trust-extracted:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  registry-certificates:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      image-registry-certificates
    Optional:  false
  trusted-ca:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      trusted-ca
    Optional:  true
  installation-pull-secrets:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  installation-pull-secrets
    Optional:    true
  bound-sa-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3600
  kube-api-access-hgps4:
    Type:                     Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:   3607
    ConfigMapName:            kube-root-ca.crt
    ConfigMapOptional:        <nil>
    DownwardAPI:              true
    ConfigMapName:            openshift-service-ca.crt
    ConfigMapOptional:        <nil>
QoS Class:                    Burstable
Node-Selectors:               kubernetes.io/os=linux
Tolerations:                  node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                              node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                              node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints:  kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
                              node-role.kubernetes.io/worker:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
Events:
  Type     Reason            Age                      From               Message
  ----     ------            ----                     ----               -------
  Warning  FailedScheduling  4m25s (x421 over 2d23h)  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
praveenkumar commented 1 year ago

@ghatwala the image registry pod is in pending because of https://github.com/crc-org/snc/blob/master/snc-library.sh#L181-L186, as part of snc we also create persistent volume for image registry using hostpath CSI driver and since that is not available it is not able to create the PVC which registry pod required :( so you might delete the https://github.com/crc-org/snc/blob/master/registry_pvc.yaml resource manually and also revert using pvc to emptyDir.

ghatwala commented 1 year ago

thanks @praveenkumar for above workaround , what i did was after i deleted the PVC using ./openshift-clients/linux/oc delete -f registry_pvc.yaml , updated the deployment -"image-registry " by changing the volume -"registry-storage" to have emptyDir.instead of PersistentvolumeClaim

- emptyDir: {}
  name: registry-storage

It has storage class which will still not work due to missing power image.

# ./openshift-clients/linux/oc get sc
NAME                                     PROVISIONER                        RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
crc-csi-hostpath-provisioner (default)   kubevirt.io.hostpath-provisioner   Delete          WaitForFirstConsumer   false                  3d20h

However in "openshift-image-registry" NS , am getting all resources as up

# ./openshift-clients/linux/oc get all -n openshift-image-registry
NAME                                                  READY   STATUS      RESTARTS   AGE
pod/cluster-image-registry-operator-df49cfd7f-mwpkt   1/1     Running     2          4d21h
pod/image-pruner-28097280-qjszz                       0/1     Completed   0          2d10h
pod/image-pruner-28098720-k8mwz                       0/1     Completed   0          34h
pod/image-pruner-28100160-6fhcc                       0/1     Completed   0          10h
pod/image-registry-59bc55d7cd-2n8rn                   1/1     Running     0          13m
pod/node-ca-kcvhs                                     1/1     Running     2          4d21h

NAME                              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
service/image-registry            ClusterIP   10.217.4.110   <none>        5000/TCP    4d21h
service/image-registry-operator   ClusterIP   None           <none>        60000/TCP   4d21h

NAME                     DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/node-ca   1         1         1       1            1           kubernetes.io/os=linux   4d21h

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cluster-image-registry-operator   1/1     1            1           4d21h
deployment.apps/image-registry                    1/1     1            1           13m

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/cluster-image-registry-operator-df49cfd7f   1         1         1       4d21h
replicaset.apps/image-registry-59bc55d7cd                   1         1         1       13m

NAME                         SCHEDULE    SUSPEND   ACTIVE   LAST SCHEDULE   AGE
cronjob.batch/image-pruner   0 0 * * *   False     0        10h             4d21h

NAME                              COMPLETIONS   DURATION   AGE
job.batch/image-pruner-28097280   1/1           9s         2d10h
job.batch/image-pruner-28098720   1/1           13s        34h
job.batch/image-pruner-28100160   1/1           12s        10h

NAME                                     HOST/PORT                                                 PATH   SERVICES         PORT    TERMINATION   WILDCARD
route.route.openshift.io/default-route   default-route-openshift-image-registry.apps-crc.testing          image-registry   <all>   reencrypt     None

However the image-registry CO is showing as "progressing"

# ./openshift-clients/linux/oc get co image-registry
NAME             VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
image-registry   4.13.0-ec.2   True        True          False      19m     Progressing: Unable to apply resources: unable to sync storage configuration: persistentvolumeclaims "crc-image-registry-storage" not found...

However cluster seems to be up

# ./openshift-clients/linux/oc get nodes
NAME                 STATUS   ROLES                         AGE     VERSION
crc-ks9kv-master-0   Ready    control-plane,master,worker   4d21h   v1.25.2+7dab57f

# ./openshift-clients/linux/oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-ec.2   True        False         4d21h   Cluster version is 4.13.0-ec.2

Another side query - is anyone working on enabling power support for this image - "https://quay.io/repository/crcont/hostpath-csi-driver?tab=tags&tag=latest" guess this image will block smooth installation of SNC on power arches ?

cfergeau commented 1 year ago

You could try to workaround these problems by reusing the old way of creating PVs, maybe this still works well enough to allow the bundle generation to succeed on ppc https://github.com/crc-org/snc/commit/77aa37ee5a74b3cff7506bae3b53292d028073f9#diff-ef06be85fabe62df5f44fe49c0cf0728efd4b824c768a168e6a385ae220e39b3

praveenkumar commented 1 year ago

Another side query - is anyone working on enabling power support for this image - "https://quay.io/repository/crcont/hostpath-csi-driver?tab=tags&tag=latest" guess this image will block smooth installation of SNC on power arches ?

@mtarsel was trying out snc on PPC not sure about if he worked on creating hostpath-csi-driver image for ppc.

ghatwala commented 1 year ago

hey @cfergeau . updates on your below input.

You could try to workaround these problems by reusing the old way of creating PVs, maybe this still works well enough to allow the bundle generation to succeed on ppc https://github.com/crc-org/snc/commit/77aa37ee5a74b3cff7506bae3b53292d028073f9#diff-ef06be85fabe62df5f44fe49c0cf0728efd4b824c768a168e6a385ae220e39b3

Using just relevant parts of code in this in latest code and creating pv using hostPath instead of storage class helped.

# ./openshift-clients/linux/oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-ec.2   True        False         8d      Cluster version is 4.13.0-ec.2

# ./openshift-clients/linux/oc get co
NAME                                       VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.0-ec.2   True        False         False      8m3s
config-operator                            4.13.0-ec.2   True        False         False      8d
console                                    4.13.0-ec.2   True        False         False      8d
control-plane-machine-set                  4.13.0-ec.2   True        False         False      8d
dns                                        4.13.0-ec.2   True        False         False      8d
etcd                                       4.13.0-ec.2   True        False         False      8d
image-registry                             4.13.0-ec.2   True        False         False      8d
ingress                                    4.13.0-ec.2   True        False         False      8d
kube-apiserver                             4.13.0-ec.2   True        False         False      8d
kube-controller-manager                    4.13.0-ec.2   True        False         False      8d
kube-scheduler                             4.13.0-ec.2   True        False         False      8d
kube-storage-version-migrator              4.13.0-ec.2   True        False         False      8d
machine-api                                4.13.0-ec.2   True        False         False      8d
machine-approver                           4.13.0-ec.2   True        False         False      8d
machine-config                             4.13.0-ec.2   True        False         False      8d
marketplace                                4.13.0-ec.2   True        False         False      8d
network                                    4.13.0-ec.2   True        False         False      8d
node-tuning                                4.13.0-ec.2   True        False         False      8d
openshift-apiserver                        4.13.0-ec.2   True        False         False      8d
openshift-controller-manager               4.13.0-ec.2   True        False         False      7m51s
openshift-samples                          4.13.0-ec.2   True        False         False      8d
operator-lifecycle-manager                 4.13.0-ec.2   True        False         False      8d
operator-lifecycle-manager-catalog         4.13.0-ec.2   True        False         False      8d
operator-lifecycle-manager-packageserver   4.13.0-ec.2   True        False         False      10m
service-ca                                 4.13.0-ec.2   True        False         False      8d

# ./openshift-clients/linux/oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.0-ec.2   True        False         8d      Cluster version is 4.13.0-ec.2

# ./openshift-clients/linux/oc get nodes
NAME                 STATUS   ROLES                         AGE   VERSION
crc-h45qx-master-0   Ready    control-plane,master,worker   8d    v1.25.2+7dab57f

@praveenkumar - as regards to "creating hostpath-csi-driver image for ppc64le." , will try it sometime soon hopefully.

ghatwala commented 1 year ago

hey @praveenkumar / @cfergeau

Noticed that machine-config CO went degraded soon after ( this was using just relevant parts of code in this in latest code and creating pv using hostPath instead )

# ./openshift-clients/linux/oc get co machine-config
NAME             VERSION       AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
machine-config   4.13.0-ec.2   True        False         True       3d19h   Failed to resync 4.13.0-ec.2 because: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 0)]

Describing the CO.

# ./openshift-clients/linux/oc describe co machine-config
Name:         machine-config
Namespace:
Labels:       <none>
Annotations:  exclude.release.openshift.io/internal-openshift-hosted: true
              include.release.openshift.io/self-managed-high-availability: true
              include.release.openshift.io/single-node-developer: true
API Version:  config.openshift.io/v1
Kind:         ClusterOperator
Metadata:
  Creation Timestamp:  2023-06-08T10:45:11Z
  Generation:          1
  Managed Fields:
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:exclude.release.openshift.io/internal-openshift-hosted:
          f:include.release.openshift.io/self-managed-high-availability:
          f:include.release.openshift.io/single-node-developer:
        f:ownerReferences:
          .:
          k:{"uid":"75a3f382-8a53-471d-82a1-843ab44cc4f9"}:
      f:spec:
    Manager:      cluster-version-operator
    Operation:    Update
    Time:         2023-06-08T10:45:11Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
    Manager:      cluster-version-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-06-08T10:45:11Z
    API Version:  config.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
        f:extension:
          .:
          f:master:
          f:worker:
        f:relatedObjects:
        f:versions:
    Manager:      machine-config-operator
    Operation:    Update
    Subresource:  status
    Time:         2023-06-12T06:39:40Z
  Owner References:
    API Version:     config.openshift.io/v1
    Kind:            ClusterVersion
    Name:            version
    UID:             75a3f382-8a53-471d-82a1-843ab44cc4f9
  Resource Version:  1352992
  UID:               c4d47c05-294d-4d23-97f6-8ea886439d34
Spec:
Status:
  Conditions:
    Last Transition Time:  2023-06-08T10:48:53Z
    Message:               Cluster version is 4.13.0-ec.2
    Status:                False
    Type:                  Progressing
    Last Transition Time:  2023-06-09T11:43:22Z
    Message:               One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading
    Reason:                DegradedPool
    Status:                False
    Type:                  Upgradeable
    Last Transition Time:  2023-06-09T11:53:21Z
    Message:               Failed to resync 4.13.0-ec.2 because: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 0)]
    Reason:                RequiredPoolsFailed
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2023-06-08T10:48:52Z
    Message:               Cluster has deployed [{operator 4.13.0-ec.2}]
    Reason:                AsExpected
    Status:                True
    Type:                  Available
  Extension:
    Master:  pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node crc-2ctjr-master-0 is reporting: \"prepping update: machineconfig.machineconfiguration.openshift.io \\\"rendered-master-b2614334c19dc9f335083585765ea9d0\\\" not found\""
    Worker:  all 0 nodes are at latest configuration rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327
  Related Objects:
    Group:
    Name:      openshift-machine-config-operator
    Resource:  namespaces
    Group:     machineconfiguration.openshift.io
    Name:
    Resource:  machineconfigpools
    Group:     machineconfiguration.openshift.io
    Name:
    Resource:  controllerconfigs
    Group:     machineconfiguration.openshift.io
    Name:
    Resource:  kubeletconfigs
    Group:     machineconfiguration.openshift.io
    Name:
    Resource:  containerruntimeconfigs
    Group:     machineconfiguration.openshift.io
    Name:
    Resource:  machineconfigs
    Group:
    Name:
    Resource:  nodes
    Group:
    Name:      openshift-kni-infra
    Resource:  namespaces
    Group:
    Name:      openshift-openstack-infra
    Resource:  namespaces
    Group:
    Name:      openshift-ovirt-infra
    Resource:  namespaces
    Group:
    Name:      openshift-vsphere-infra
    Resource:  namespaces
    Group:
    Name:      openshift-nutanix-infra
    Resource:  namespaces
  Versions:
    Name:     operator
    Version:  4.13.0-ec.2
Events:       <none>

Checking mcp

# ./openshift-clients/linux/oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-b2614334c19dc9f335083585765ea9d0   False     True       True       1              0                   0                     1                      3d19h
worker   rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327   True      False      False      0              0                   0                     0                      3d19h

Describing the mcp ...

# ./openshift-clients/linux/oc describe mcp master
Name:         master
Namespace:
Labels:       machineconfiguration.openshift.io/mco-built-in=
              operator.machineconfiguration.openshift.io/required-for-upgrade=
              pools.operator.machineconfiguration.openshift.io/master=
Annotations:  <none>
API Version:  machineconfiguration.openshift.io/v1
Kind:         MachineConfigPool
Metadata:
  Creation Timestamp:  2023-06-08T10:47:30Z
  Generation:          6
  Managed Fields:
    API Version:  machineconfiguration.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .:
          f:machineconfiguration.openshift.io/mco-built-in:
          f:operator.machineconfiguration.openshift.io/required-for-upgrade:
          f:pools.operator.machineconfiguration.openshift.io/master:
      f:spec:
        .:
        f:configuration:
        f:machineConfigSelector:
          .:
          f:matchLabels:
            .:
            f:machineconfiguration.openshift.io/role:
        f:nodeSelector:
          .:
          f:matchLabels:
            .:
            f:node-role.kubernetes.io/master:
        f:paused:
    Manager:      machine-config-operator
    Operation:    Update
    Time:         2023-06-08T10:47:30Z
    API Version:  machineconfiguration.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        f:configuration:
          f:name:
          f:source:
    Manager:      machine-config-controller
    Operation:    Update
    Time:         2023-06-09T11:43:17Z
    API Version:  machineconfiguration.openshift.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:configuration:
          .:
          f:name:
          f:source:
        f:degradedMachineCount:
        f:machineCount:
        f:observedGeneration:
        f:readyMachineCount:
        f:unavailableMachineCount:
        f:updatedMachineCount:
    Manager:         machine-config-controller
    Operation:       Update
    Subresource:     status
    Time:            2023-06-09T12:03:37Z
  Resource Version:  33867
  UID:               a1a4b5cc-65bc-4466-9087-15b2cb5c108d
Spec:
  Configuration:
    Name:  rendered-master-d0d5dc695ecc939aa643095cc77a2219
    Source:
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         00-master
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-master-container-runtime
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         01-master-kubelet
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-master-generated-registries
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-master-ssh
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-node-sizing-for-crc
      API Version:  machineconfiguration.openshift.io/v1
      Kind:         MachineConfig
      Name:         99-openshift-machineconfig-master-dummy-networks
  Machine Config Selector:
    Match Labels:
      machineconfiguration.openshift.io/role:  master
  Node Selector:
    Match Labels:
      node-role.kubernetes.io/master:
  Paused:                              false
Status:
  Conditions:
    Last Transition Time:  2023-06-08T10:48:16Z
    Message:
    Reason:
    Status:                False
    Type:                  RenderDegraded
    Last Transition Time:  2023-06-09T11:43:22Z
    Message:
    Reason:
    Status:                False
    Type:                  Updated
    Last Transition Time:  2023-06-09T11:43:22Z
    Message:               All nodes are updating to rendered-master-d0d5dc695ecc939aa643095cc77a2219
    Reason:
    Status:                True
    Type:                  Updating
    Last Transition Time:  2023-06-09T11:43:40Z
    Message:
    Reason:
    Status:                True
    Type:                  Degraded
    Last Transition Time:  2023-06-09T11:43:40Z
    Message:               Node crc-2ctjr-master-0 is reporting: "prepping update: machineconfig.machineconfiguration.openshift.io \"rendered-master-b2614334c19dc9f335083585765ea9d0\" not found"
    Reason:                1 nodes are reporting degraded status on sync
    Status:                True
    Type:                  NodeDegraded
  Configuration:
    Name:  rendered-master-b2614334c19dc9f335083585765ea9d0
    Source:
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   00-master
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-master-container-runtime
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   01-master-kubelet
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-master-generated-registries
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-master-ssh
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-node-sizing-for-crc
      API Version:            machineconfiguration.openshift.io/v1
      Kind:                   MachineConfig
      Name:                   99-openshift-machineconfig-master-dummy-networks
  Degraded Machine Count:     1
  Machine Count:              1
  Observed Generation:        6
  Ready Machine Count:        0
  Unavailable Machine Count:  0
  Updated Machine Count:      0
Events:                       <none>

Logs for snc.sh ..

+ echo 'Cluster has stabilized'
Cluster has stabilized
+ retry ./openshift-clients/linux/oc delete pods --field-selector=status.phase=Failed -A
+ local retries=10
+ local count=0
+ ./openshift-clients/linux/oc delete pods --field-selector=status.phase=Failed -A
No resources found
+ return 0
+ break
+ retry all_pods_are_running_completed openshift-marketplace
+ local retries=10
+ local count=0
+ all_pods_are_running_completed openshift-marketplace
+ local ignoreNamespace=openshift-marketplace
+ ./openshift-clients/linux/oc get pod --no-headers --all-namespaces '--field-selector=metadata.namespace!=openshift-marketplace'
+ grep -v Running
+ grep -v Completed
+ return 0
+ retry ./openshift-clients/linux/oc delete pod --field-selector=status.phase==Succeeded --all-namespaces
+ local retries=10
+ local count=0
+ ./openshift-clients/linux/oc delete pod --field-selector=status.phase==Succeeded --all-namespaces
pod "installer-3-crc-2ctjr-master-0" deleted
pod "installer-4-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-10-crc-2ctjr-master-0" deleted
pod "installer-2-crc-2ctjr-master-0" deleted
pod "installer-4-retry-1-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-6-crc-2ctjr-master-0" deleted
pod "installer-8-crc-2ctjr-master-0" deleted
pod "installer-5-retry-1-crc-2ctjr-master-0" deleted
pod "installer-6-retry-1-crc-2ctjr-master-0" deleted
pod "installer-7-crc-2ctjr-master-0" deleted
pod "revision-pruner-7-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-6-crc-2ctjr-master-0" deleted
pod "revision-pruner-6-crc-2ctjr-master-0" deleted
pod "collect-profiles-28105155-lxgdb" deleted
pod "collect-profiles-28105170-xh72n" deleted
pod "collect-profiles-28105185-wwgx4" deleted
+ return 0
++ retry ./openshift-clients/linux/oc get mc --sort-by=.metadata.creationTimestamp --no-headers -oname
++ local retries=10
++ local count=0
++ ./openshift-clients/linux/oc get mc --sort-by=.metadata.creationTimestamp --no-headers -oname
++ return 0
+ mc_name='machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ echo 'machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ grep rendered-master
+ head -n -1
+ xargs -t ./openshift-clients/linux/oc delete
./openshift-clients/linux/oc delete machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78 machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2 machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37 machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0 
machineconfig.machineconfiguration.openshift.io "rendered-master-106dd41a9320b2f12717cabdb52fbc78" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-82f2facd6871b5ae714d735ad4a9a5b2" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-f67e28e8e1e4b1e223f544aa125c9d37" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-b2614334c19dc9f335083585765ea9d0" deleted
+ echo 'machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ grep rendered-worker
+ head -n -1
+ xargs -t ./openshift-clients/linux/oc delete
./openshift-clients/linux/oc delete machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283 machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97 machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d 
machineconfig.machineconfiguration.openshift.io "rendered-worker-3559683dc13f9a32a1c7f82c1761a283" deleted
machineconfig.machineconfiguration.openshift.io "rendered-worker-7e0440ebe6381621e527a773a4f42d97" deleted
machineconfig.machineconfiguration.openshift.io "rendered-worker-32ca10bba66026336f26ca8794c3826d" deleted
+ jobs=($(jobs -p))
++ jobs -p
+ '[' -n '' ']'
+ true
cfergeau commented 1 year ago

Why are you trying 4.13.0.ec.2? latest ppc64le build is 4.13.2 https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp/latest-4.13/

ghatwala commented 1 year ago

Why are you trying 4.13.0.ec.2? latest ppc64le build is 4.13.2 https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp/latest-4.13/

as that's the value we get by default snc.sh from here - https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/latest-4.13/release.txt from here - https://github.com/crc-org/snc/blob/master/snc.sh#L34

check logs below . 

export LC_ALL=C.UTF-8
+ LC_ALL=C.UTF-8
+ export LANG=C.UTF-8
+ LANG=C.UTF-8
+ source tools.sh
++ JQ=jq
++ QEMU_IMG=qemu-img
++ VIRT_FILESYSTEMS=virt-filesystems
++ GUESTFISH=guestfish
++ VIRSH=virsh
++ XMLLINT=xmllint
++ DIG=dig
++ UNZIP=unzip
++ ZSTD=zstd
++ CRC_ZSTD_EXTRA_FLAGS='--ultra -22'
++ HTPASSWD=htpasswd
++ PATCH=patch
+++ uname -m
++ ARCH=ppc64le
++ case "${ARCH}" in
++ yq_ARCH=ppc64le
++ SNC_GENERATE_MACOS_BUNDLE=0
++ SNC_GENERATE_WINDOWS_BUNDLE=0
++ SNC_GENERATE_LINUX_BUNDLE=1
++ test -z
++ echo 'Downloading yq binary to manipulate yaml files'
Downloading yq binary to manipulate yaml files
++ curl -L https://github.com/mikefarah/yq/releases/download/v4.5.1/yq_linux_ppc64le -o yq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

100 6599k  100 6599k    0     0  16.6M      0 --:--:-- --:--:-- --:--:-- 16.6M
++ chmod +x yq
++ YQ=./yq
++ which jq
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot jq
/usr/bin/jq
++ which virt-filesystems
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot virt-filesystems
++ alias
++ eval declare -f
+++ declare -f
/usr/bin/virt-filesystems
++ which guestfish
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot guestfish
/usr/bin/guestfish
++ which virsh
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot virsh
/usr/bin/virsh
++ which qemu-img
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot qemu-img
/usr/bin/qemu-img
++ rpm -q libguestfs-xfs
libguestfs-xfs-1.44.0-9.module+el8.8.0+16781+9f4724c2.ppc64le
++ '[' 0 '!=' 0 -o 0 '!=' 0 ']'
++ which xmllint
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot xmllint
/usr/bin/xmllint
++ which dig
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot dig
/usr/bin/dig
++ which zstd
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot zstd
/usr/bin/zstd
++ which htpasswd
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot htpasswd
/usr/bin/htpasswd
++ which patch
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot patch
/usr/bin/patch
+ source snc-library.sh
++ set -exuo pipefail
+ trap 'jobs=($(jobs -p)); [ -n "${jobs-}" ] && ((${#jobs})) && kill "${jobs[@]}" || true' EXIT
+ OKD_VERSION=none
+ BUNDLE_TYPE=snc
+ [[ none != \n\o\n\e ]]
+ INSTALL_DIR=crc-tmp-install-data
+ SNC_PRODUCT_NAME=crc
+ SNC_CLUSTER_MEMORY=14336
+ SNC_CLUSTER_CPUS=6
+ CRC_VM_DISK_SIZE=33285996544
+ BASE_DOMAIN=testing
+ CRC_PV_DIR=/mnt/pv-data
+ SSH='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ SCP='scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ MIRROR=https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview
+ CERT_ROTATION=enabled
+ USE_PATCHED_RELEASE_IMAGE=disabled
+ HTPASSWD_FILE=users.htpasswd
+ run_preflight_checks snc
+ local bundle_type=snc
+ '[' -z /home/amit_snc/pull_secret.json ']'
+ '[' '!' -f /home/amit_snc/pull_secret.json ']'
+ echo 'Checking libvirt and DNS configuration'
Checking libvirt and DNS configuration
+ LIBVIRT_URI=qemu:///system
+ '[' snc == snc ']'
+ LIBVIRT_URI=qemu+tcp://localhost/system
+ virsh -c qemu+tcp://localhost/system uri
+ virsh -c qemu+tcp://localhost/system net-info default
+ echo 'default libvirt network is available'
default libvirt network is available
++ virsh -c qemu+tcp://localhost/system net-info default
++ awk '{print $2}'
++ sed '3q;d'
+ [[ yes == \n\o ]]
+ case $ARCH in
+ echo 'The host arch is ppc64le.'
The host arch is ppc64le.
+ virsh -c qemu+tcp://localhost/system capabilities
+ xmllint --xpath '/capabilities/guest/arch[@name='\''ppc64le'\'']/domain[@type='\''kvm'\'']' -
+ '[' snc == snc ']'
+ local ping_status
++ ping -c1 api.crc.testing
++ head -1
+ ping_status='PING api.crc.testing (192.168.126.11) 56(84) bytes of data.'
+ grep 'PING api.crc.testing ('
+ echo PING api.crc.testing '(192.168.126.11)' '56(84)' bytes of data.
PING api.crc.testing (192.168.126.11) 56(84) bytes of data.
+ echo PING api.crc.testing '(192.168.126.11)' '56(84)' bytes of data.
+ grep '192.168.126.1[01])'
PING api.crc.testing (192.168.126.11) 56(84) bytes of data.
+ echo 'libvirt and DNS configuration successfully checked'
libvirt and DNS configuration successfully checked
+ test -n ''
++ curl -L https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/latest-4.13/release.txt
++ sed -n 's/^ *Version: *//p'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 31548  100 31548    0     0   405k      0 --:--:-- --:--:-- --:--:--  405k
+ OPENSHIFT_RELEASE_VERSION=4.13.0-ec.2
ghatwala commented 1 year ago

Using latest 4.13 via below change , thanks @cfergeau !!!

-MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp-dev-preview}
+MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp}

getting better results .

[root@ltc22briggs020 snc]# ./openshift-clients/linux/oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.2    True        False         24h     Cluster version is 4.13.2
[root@ltc22briggs020 snc]# ./openshift-clients/linux/oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.13.2    True        False         False      33m
config-operator                            4.13.2    True        False         False      25h
console                                    4.13.2    True        False         False      24m
control-plane-machine-set                  4.13.2    True        False         False      25h
dns                                        4.13.2    True        False         False      39m
etcd                                       4.13.2    True        False         False      25h
image-registry                             4.13.2    True        False         False      24h
ingress                                    4.13.2    True        False         False      25h
kube-apiserver                             4.13.2    True        False         False      24h
kube-controller-manager                    4.13.2    True        False         False      24h
kube-scheduler                             4.13.2    True        False         False      24h
kube-storage-version-migrator              4.13.2    True        False         False      25h
machine-api                                4.13.2    True        False         False      24h
machine-approver                           4.13.2    True        False         False      25h
machine-config                             4.13.2    True        False         False      25h
marketplace                                4.13.2    True        False         False      25h
network                                    4.13.2    True        False         False      25h
openshift-apiserver                        4.13.2    True        False         False      24m
openshift-controller-manager               4.13.2    True        False         False      34m
openshift-samples                          4.13.2    True        False         False      24h
operator-lifecycle-manager                 4.13.2    True        False         False      25h
operator-lifecycle-manager-catalog         4.13.2    True        False         False      25h
operator-lifecycle-manager-packageserver   4.13.2    True        False         False      29m
service-ca                                 4.13.2    True        False         False      25h
cfergeau commented 1 year ago

Using latest 4.13 via below change , thanks @cfergeau !!!

-MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp-dev-preview}
+MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp}

This was a bug in snc, we forgot to update it, thanks for pointing it out! It's fixed in master now: https://github.com/crc-org/snc/commit/6575a3c552438be76e73c4feb2325752912c4002

praveenkumar commented 1 year ago

@ghatwala should we close this issue now?

ghatwala commented 1 year ago

Yes , closing this issue !!!