Closed ghatwala closed 1 year ago
@ghatwala Problem with hostpath-csi-driver
because the image we have is only for amd64 and arm64 arch not for power pc arch :( other than that I don't see any issue and if you don't need CSI driver, this can be ignored.
thanks @praveenkumar for your response and ack on ignoring the hostpath-csi-driver
based pod , however in my logs above there's a cluster operator ( image-registry
) which is failing/degraded state ( pod is in pending state) , capturing info again , any inputs/pointers on how to get this CO in true state.
image-registry 4.13.0-ec.2 True True True 4d Degraded: Registry deployment has timed out progressing: ReplicaSet "image-registry-76db78f4d5" has timed out progressing.
# ./openshift-clients/linux/oc get po -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
cluster-image-registry-operator-df49cfd7f-mwpkt 1/1 Running 2 4d1h
image-pruner-28095840-wg4l2 0/1 Completed 0 2d13h
image-pruner-28097280-qjszz 0/1 Completed 0 37h
image-pruner-28098720-k8mwz 0/1 Completed 0 13h
image-registry-76db78f4d5-xvp8n 0/1 Pending 0 2d23h
image-registry-d7f77b585-mghtw 1/1 Running 2 4d
node-ca-kcvhs 1/1 Running 2 4d
checking failing pod logs
# ./openshift-clients/linux/oc describe po -n openshift-image-registry image-registry-76db78f4d5-xvp8n
Name: image-registry-76db78f4d5-xvp8n
Namespace: openshift-image-registry
Priority: 2000000000
Priority Class Name: system-cluster-critical
Service Account: registry
Node: <none>
Labels: docker-registry=default
pod-template-hash=76db78f4d5
Annotations: imageregistry.operator.openshift.io/dependencies-checksum: sha256:5851fee4a9fb3ce4403dbd14db0c8993ad35242c92fdd0a79f3f09722d545638
openshift.io/scc: restricted-v2
seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/image-registry-76db78f4d5
Containers:
registry:
Image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7e01095809c29cbd68b1c3c3c4a5c0b249e4fcb9d642a80f9405a208f0ee257
Port: 5000/TCP
Host Port: 0/TCP
Command:
/bin/sh
-c
mkdir -p /etc/pki/ca-trust/extracted/edk2 /etc/pki/ca-trust/extracted/java /etc/pki/ca-trust/extracted/openssl /etc/pki/ca-trust/extracted/pem && update-ca-trust extract && exec /usr/bin/dockerregistry
Requests:
cpu: 100m
memory: 256Mi
Liveness: http-get https://:5000/healthz delay=5s timeout=5s period=10s #success=1 #failure=3
Readiness: http-get https://:5000/healthz delay=15s timeout=5s period=10s #success=1 #failure=3
Environment:
REGISTRY_STORAGE: filesystem
REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /registry
REGISTRY_HTTP_ADDR: :5000
REGISTRY_HTTP_NET: tcp
REGISTRY_HTTP_SECRET: a16ee665091727a6e4fa49d9ac55cf62f56683c17c806380e9ca8204f00cc57878861f3e149aab984ec6cce4198427ab0ff8fd4f1e56aa01f90ea9806d8c1950
REGISTRY_LOG_LEVEL: info
REGISTRY_OPENSHIFT_QUOTA_ENABLED: true
REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR: inmemory
REGISTRY_STORAGE_DELETE_ENABLED: true
REGISTRY_HEALTH_STORAGEDRIVER_ENABLED: true
REGISTRY_HEALTH_STORAGEDRIVER_INTERVAL: 10s
REGISTRY_HEALTH_STORAGEDRIVER_THRESHOLD: 1
REGISTRY_OPENSHIFT_METRICS_ENABLED: true
REGISTRY_OPENSHIFT_SERVER_ADDR: image-registry.openshift-image-registry.svc:5000
REGISTRY_HTTP_TLS_CERTIFICATE: /etc/secrets/tls.crt
REGISTRY_HTTP_TLS_KEY: /etc/secrets/tls.key
Mounts:
/etc/pki/ca-trust/extracted from ca-trust-extracted (rw)
/etc/pki/ca-trust/source/anchors from registry-certificates (rw)
/etc/secrets from registry-tls (rw)
/registry from registry-storage (rw)
/usr/share/pki/ca-trust-source from trusted-ca (rw)
/var/lib/kubelet/ from installation-pull-secrets (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hgps4 (ro)
/var/run/secrets/openshift/serviceaccount from bound-sa-token (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
registry-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: crc-image-registry-storage
ReadOnly: false
registry-tls:
Type: Projected (a volume that contains injected data from multiple sources)
SecretName: image-registry-tls
SecretOptionalName: <nil>
ca-trust-extracted:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
registry-certificates:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: image-registry-certificates
Optional: false
trusted-ca:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: trusted-ca
Optional: true
installation-pull-secrets:
Type: Secret (a volume populated by a Secret)
SecretName: installation-pull-secrets
Optional: true
bound-sa-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3600
kube-api-access-hgps4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
ConfigMapName: openshift-service-ca.crt
ConfigMapOptional: <nil>
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Topology Spread Constraints: kubernetes.io/hostname:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
node-role.kubernetes.io/worker:DoNotSchedule when max skew 1 is exceeded for selector docker-registry=default
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 4m25s (x421 over 2d23h) default-scheduler running PreBind plugin "VolumeBinding": binding volumes: timed out waiting for the condition
@ghatwala the image registry pod is in pending because of https://github.com/crc-org/snc/blob/master/snc-library.sh#L181-L186, as part of snc we also create persistent volume for image registry using hostpath CSI driver and since that is not available it is not able to create the PVC which registry pod required :( so you might delete the https://github.com/crc-org/snc/blob/master/registry_pvc.yaml resource manually and also revert using pvc to emptyDir.
thanks @praveenkumar for above workaround , what i did was after i deleted the PVC using ./openshift-clients/linux/oc delete -f registry_pvc.yaml
, updated the deployment -"image-registry " by changing the volume -"registry-storage" to have emptyDir.instead of PersistentvolumeClaim
- emptyDir: {}
name: registry-storage
It has storage class which will still not work due to missing power image.
# ./openshift-clients/linux/oc get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
crc-csi-hostpath-provisioner (default) kubevirt.io.hostpath-provisioner Delete WaitForFirstConsumer false 3d20h
However in "openshift-image-registry" NS , am getting all resources as up
# ./openshift-clients/linux/oc get all -n openshift-image-registry
NAME READY STATUS RESTARTS AGE
pod/cluster-image-registry-operator-df49cfd7f-mwpkt 1/1 Running 2 4d21h
pod/image-pruner-28097280-qjszz 0/1 Completed 0 2d10h
pod/image-pruner-28098720-k8mwz 0/1 Completed 0 34h
pod/image-pruner-28100160-6fhcc 0/1 Completed 0 10h
pod/image-registry-59bc55d7cd-2n8rn 1/1 Running 0 13m
pod/node-ca-kcvhs 1/1 Running 2 4d21h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/image-registry ClusterIP 10.217.4.110 <none> 5000/TCP 4d21h
service/image-registry-operator ClusterIP None <none> 60000/TCP 4d21h
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/node-ca 1 1 1 1 1 kubernetes.io/os=linux 4d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/cluster-image-registry-operator 1/1 1 1 4d21h
deployment.apps/image-registry 1/1 1 1 13m
NAME DESIRED CURRENT READY AGE
replicaset.apps/cluster-image-registry-operator-df49cfd7f 1 1 1 4d21h
replicaset.apps/image-registry-59bc55d7cd 1 1 1 13m
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
cronjob.batch/image-pruner 0 0 * * * False 0 10h 4d21h
NAME COMPLETIONS DURATION AGE
job.batch/image-pruner-28097280 1/1 9s 2d10h
job.batch/image-pruner-28098720 1/1 13s 34h
job.batch/image-pruner-28100160 1/1 12s 10h
NAME HOST/PORT PATH SERVICES PORT TERMINATION WILDCARD
route.route.openshift.io/default-route default-route-openshift-image-registry.apps-crc.testing image-registry <all> reencrypt None
However the image-registry CO is showing as "progressing"
# ./openshift-clients/linux/oc get co image-registry
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
image-registry 4.13.0-ec.2 True True False 19m Progressing: Unable to apply resources: unable to sync storage configuration: persistentvolumeclaims "crc-image-registry-storage" not found...
However cluster seems to be up
# ./openshift-clients/linux/oc get nodes
NAME STATUS ROLES AGE VERSION
crc-ks9kv-master-0 Ready control-plane,master,worker 4d21h v1.25.2+7dab57f
# ./openshift-clients/linux/oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.0-ec.2 True False 4d21h Cluster version is 4.13.0-ec.2
Another side query - is anyone working on enabling power support for this image - "https://quay.io/repository/crcont/hostpath-csi-driver?tab=tags&tag=latest" guess this image will block smooth installation of SNC on power arches ?
You could try to workaround these problems by reusing the old way of creating PVs, maybe this still works well enough to allow the bundle generation to succeed on ppc https://github.com/crc-org/snc/commit/77aa37ee5a74b3cff7506bae3b53292d028073f9#diff-ef06be85fabe62df5f44fe49c0cf0728efd4b824c768a168e6a385ae220e39b3
Another side query - is anyone working on enabling power support for this image - "https://quay.io/repository/crcont/hostpath-csi-driver?tab=tags&tag=latest" guess this image will block smooth installation of SNC on power arches ?
@mtarsel was trying out snc on PPC not sure about if he worked on creating hostpath-csi-driver image for ppc.
hey @cfergeau . updates on your below input.
You could try to workaround these problems by reusing the old way of creating PVs, maybe this still works well enough to allow the bundle generation to succeed on ppc https://github.com/crc-org/snc/commit/77aa37ee5a74b3cff7506bae3b53292d028073f9#diff-ef06be85fabe62df5f44fe49c0cf0728efd4b824c768a168e6a385ae220e39b3
Using just relevant parts of code in this in latest code and creating pv using hostPath
instead of storage class helped.
# ./openshift-clients/linux/oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.0-ec.2 True False 8d Cluster version is 4.13.0-ec.2
# ./openshift-clients/linux/oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.13.0-ec.2 True False False 8m3s
config-operator 4.13.0-ec.2 True False False 8d
console 4.13.0-ec.2 True False False 8d
control-plane-machine-set 4.13.0-ec.2 True False False 8d
dns 4.13.0-ec.2 True False False 8d
etcd 4.13.0-ec.2 True False False 8d
image-registry 4.13.0-ec.2 True False False 8d
ingress 4.13.0-ec.2 True False False 8d
kube-apiserver 4.13.0-ec.2 True False False 8d
kube-controller-manager 4.13.0-ec.2 True False False 8d
kube-scheduler 4.13.0-ec.2 True False False 8d
kube-storage-version-migrator 4.13.0-ec.2 True False False 8d
machine-api 4.13.0-ec.2 True False False 8d
machine-approver 4.13.0-ec.2 True False False 8d
machine-config 4.13.0-ec.2 True False False 8d
marketplace 4.13.0-ec.2 True False False 8d
network 4.13.0-ec.2 True False False 8d
node-tuning 4.13.0-ec.2 True False False 8d
openshift-apiserver 4.13.0-ec.2 True False False 8d
openshift-controller-manager 4.13.0-ec.2 True False False 7m51s
openshift-samples 4.13.0-ec.2 True False False 8d
operator-lifecycle-manager 4.13.0-ec.2 True False False 8d
operator-lifecycle-manager-catalog 4.13.0-ec.2 True False False 8d
operator-lifecycle-manager-packageserver 4.13.0-ec.2 True False False 10m
service-ca 4.13.0-ec.2 True False False 8d
# ./openshift-clients/linux/oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.0-ec.2 True False 8d Cluster version is 4.13.0-ec.2
# ./openshift-clients/linux/oc get nodes
NAME STATUS ROLES AGE VERSION
crc-h45qx-master-0 Ready control-plane,master,worker 8d v1.25.2+7dab57f
@praveenkumar - as regards to "creating hostpath-csi-driver image for ppc64le." , will try it sometime soon hopefully.
hey @praveenkumar / @cfergeau
Noticed that machine-config
CO went degraded soon after ( this was using just relevant parts of code in this in latest code and creating pv using hostPath instead )
# ./openshift-clients/linux/oc get co machine-config
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
machine-config 4.13.0-ec.2 True False True 3d19h Failed to resync 4.13.0-ec.2 because: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 0)]
Describing the CO.
# ./openshift-clients/linux/oc describe co machine-config
Name: machine-config
Namespace:
Labels: <none>
Annotations: exclude.release.openshift.io/internal-openshift-hosted: true
include.release.openshift.io/self-managed-high-availability: true
include.release.openshift.io/single-node-developer: true
API Version: config.openshift.io/v1
Kind: ClusterOperator
Metadata:
Creation Timestamp: 2023-06-08T10:45:11Z
Generation: 1
Managed Fields:
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:exclude.release.openshift.io/internal-openshift-hosted:
f:include.release.openshift.io/self-managed-high-availability:
f:include.release.openshift.io/single-node-developer:
f:ownerReferences:
.:
k:{"uid":"75a3f382-8a53-471d-82a1-843ab44cc4f9"}:
f:spec:
Manager: cluster-version-operator
Operation: Update
Time: 2023-06-08T10:45:11Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
Manager: cluster-version-operator
Operation: Update
Subresource: status
Time: 2023-06-08T10:45:11Z
API Version: config.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:conditions:
f:extension:
.:
f:master:
f:worker:
f:relatedObjects:
f:versions:
Manager: machine-config-operator
Operation: Update
Subresource: status
Time: 2023-06-12T06:39:40Z
Owner References:
API Version: config.openshift.io/v1
Kind: ClusterVersion
Name: version
UID: 75a3f382-8a53-471d-82a1-843ab44cc4f9
Resource Version: 1352992
UID: c4d47c05-294d-4d23-97f6-8ea886439d34
Spec:
Status:
Conditions:
Last Transition Time: 2023-06-08T10:48:53Z
Message: Cluster version is 4.13.0-ec.2
Status: False
Type: Progressing
Last Transition Time: 2023-06-09T11:43:22Z
Message: One or more machine config pools are degraded, please see `oc get mcp` for further details and resolve before upgrading
Reason: DegradedPool
Status: False
Type: Upgradeable
Last Transition Time: 2023-06-09T11:53:21Z
Message: Failed to resync 4.13.0-ec.2 because: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 1, ready 0, updated: 0, unavailable: 0)]
Reason: RequiredPoolsFailed
Status: True
Type: Degraded
Last Transition Time: 2023-06-08T10:48:52Z
Message: Cluster has deployed [{operator 4.13.0-ec.2}]
Reason: AsExpected
Status: True
Type: Available
Extension:
Master: pool is degraded because nodes fail with "1 nodes are reporting degraded status on sync": "Node crc-2ctjr-master-0 is reporting: \"prepping update: machineconfig.machineconfiguration.openshift.io \\\"rendered-master-b2614334c19dc9f335083585765ea9d0\\\" not found\""
Worker: all 0 nodes are at latest configuration rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327
Related Objects:
Group:
Name: openshift-machine-config-operator
Resource: namespaces
Group: machineconfiguration.openshift.io
Name:
Resource: machineconfigpools
Group: machineconfiguration.openshift.io
Name:
Resource: controllerconfigs
Group: machineconfiguration.openshift.io
Name:
Resource: kubeletconfigs
Group: machineconfiguration.openshift.io
Name:
Resource: containerruntimeconfigs
Group: machineconfiguration.openshift.io
Name:
Resource: machineconfigs
Group:
Name:
Resource: nodes
Group:
Name: openshift-kni-infra
Resource: namespaces
Group:
Name: openshift-openstack-infra
Resource: namespaces
Group:
Name: openshift-ovirt-infra
Resource: namespaces
Group:
Name: openshift-vsphere-infra
Resource: namespaces
Group:
Name: openshift-nutanix-infra
Resource: namespaces
Versions:
Name: operator
Version: 4.13.0-ec.2
Events: <none>
Checking mcp
# ./openshift-clients/linux/oc get mcp
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-b2614334c19dc9f335083585765ea9d0 False True True 1 0 0 1 3d19h
worker rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327 True False False 0 0 0 0 3d19h
Describing the mcp ...
# ./openshift-clients/linux/oc describe mcp master
Name: master
Namespace:
Labels: machineconfiguration.openshift.io/mco-built-in=
operator.machineconfiguration.openshift.io/required-for-upgrade=
pools.operator.machineconfiguration.openshift.io/master=
Annotations: <none>
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfigPool
Metadata:
Creation Timestamp: 2023-06-08T10:47:30Z
Generation: 6
Managed Fields:
API Version: machineconfiguration.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:machineconfiguration.openshift.io/mco-built-in:
f:operator.machineconfiguration.openshift.io/required-for-upgrade:
f:pools.operator.machineconfiguration.openshift.io/master:
f:spec:
.:
f:configuration:
f:machineConfigSelector:
.:
f:matchLabels:
.:
f:machineconfiguration.openshift.io/role:
f:nodeSelector:
.:
f:matchLabels:
.:
f:node-role.kubernetes.io/master:
f:paused:
Manager: machine-config-operator
Operation: Update
Time: 2023-06-08T10:47:30Z
API Version: machineconfiguration.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:configuration:
f:name:
f:source:
Manager: machine-config-controller
Operation: Update
Time: 2023-06-09T11:43:17Z
API Version: machineconfiguration.openshift.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:conditions:
f:configuration:
.:
f:name:
f:source:
f:degradedMachineCount:
f:machineCount:
f:observedGeneration:
f:readyMachineCount:
f:unavailableMachineCount:
f:updatedMachineCount:
Manager: machine-config-controller
Operation: Update
Subresource: status
Time: 2023-06-09T12:03:37Z
Resource Version: 33867
UID: a1a4b5cc-65bc-4466-9087-15b2cb5c108d
Spec:
Configuration:
Name: rendered-master-d0d5dc695ecc939aa643095cc77a2219
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-master
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-master-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-master-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-master-generated-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-master-ssh
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-node-sizing-for-crc
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-openshift-machineconfig-master-dummy-networks
Machine Config Selector:
Match Labels:
machineconfiguration.openshift.io/role: master
Node Selector:
Match Labels:
node-role.kubernetes.io/master:
Paused: false
Status:
Conditions:
Last Transition Time: 2023-06-08T10:48:16Z
Message:
Reason:
Status: False
Type: RenderDegraded
Last Transition Time: 2023-06-09T11:43:22Z
Message:
Reason:
Status: False
Type: Updated
Last Transition Time: 2023-06-09T11:43:22Z
Message: All nodes are updating to rendered-master-d0d5dc695ecc939aa643095cc77a2219
Reason:
Status: True
Type: Updating
Last Transition Time: 2023-06-09T11:43:40Z
Message:
Reason:
Status: True
Type: Degraded
Last Transition Time: 2023-06-09T11:43:40Z
Message: Node crc-2ctjr-master-0 is reporting: "prepping update: machineconfig.machineconfiguration.openshift.io \"rendered-master-b2614334c19dc9f335083585765ea9d0\" not found"
Reason: 1 nodes are reporting degraded status on sync
Status: True
Type: NodeDegraded
Configuration:
Name: rendered-master-b2614334c19dc9f335083585765ea9d0
Source:
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 00-master
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-master-container-runtime
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 01-master-kubelet
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-master-generated-registries
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-master-ssh
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-node-sizing-for-crc
API Version: machineconfiguration.openshift.io/v1
Kind: MachineConfig
Name: 99-openshift-machineconfig-master-dummy-networks
Degraded Machine Count: 1
Machine Count: 1
Observed Generation: 6
Ready Machine Count: 0
Unavailable Machine Count: 0
Updated Machine Count: 0
Events: <none>
Logs for snc.sh
..
+ echo 'Cluster has stabilized'
Cluster has stabilized
+ retry ./openshift-clients/linux/oc delete pods --field-selector=status.phase=Failed -A
+ local retries=10
+ local count=0
+ ./openshift-clients/linux/oc delete pods --field-selector=status.phase=Failed -A
No resources found
+ return 0
+ break
+ retry all_pods_are_running_completed openshift-marketplace
+ local retries=10
+ local count=0
+ all_pods_are_running_completed openshift-marketplace
+ local ignoreNamespace=openshift-marketplace
+ ./openshift-clients/linux/oc get pod --no-headers --all-namespaces '--field-selector=metadata.namespace!=openshift-marketplace'
+ grep -v Running
+ grep -v Completed
+ return 0
+ retry ./openshift-clients/linux/oc delete pod --field-selector=status.phase==Succeeded --all-namespaces
+ local retries=10
+ local count=0
+ ./openshift-clients/linux/oc delete pod --field-selector=status.phase==Succeeded --all-namespaces
pod "installer-3-crc-2ctjr-master-0" deleted
pod "installer-4-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-10-crc-2ctjr-master-0" deleted
pod "installer-2-crc-2ctjr-master-0" deleted
pod "installer-4-retry-1-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-6-crc-2ctjr-master-0" deleted
pod "installer-8-crc-2ctjr-master-0" deleted
pod "installer-5-retry-1-crc-2ctjr-master-0" deleted
pod "installer-6-retry-1-crc-2ctjr-master-0" deleted
pod "installer-7-crc-2ctjr-master-0" deleted
pod "revision-pruner-7-crc-2ctjr-master-0" deleted
pod "installer-5-crc-2ctjr-master-0" deleted
pod "installer-6-crc-2ctjr-master-0" deleted
pod "revision-pruner-6-crc-2ctjr-master-0" deleted
pod "collect-profiles-28105155-lxgdb" deleted
pod "collect-profiles-28105170-xh72n" deleted
pod "collect-profiles-28105185-wwgx4" deleted
+ return 0
++ retry ./openshift-clients/linux/oc get mc --sort-by=.metadata.creationTimestamp --no-headers -oname
++ local retries=10
++ local count=0
++ ./openshift-clients/linux/oc get mc --sort-by=.metadata.creationTimestamp --no-headers -oname
++ return 0
+ mc_name='machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ echo 'machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ grep rendered-master
+ head -n -1
+ xargs -t ./openshift-clients/linux/oc delete
./openshift-clients/linux/oc delete machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78 machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2 machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37 machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io "rendered-master-106dd41a9320b2f12717cabdb52fbc78" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-82f2facd6871b5ae714d735ad4a9a5b2" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-f67e28e8e1e4b1e223f544aa125c9d37" deleted
machineconfig.machineconfiguration.openshift.io "rendered-master-b2614334c19dc9f335083585765ea9d0" deleted
+ echo 'machineconfig.machineconfiguration.openshift.io/99-openshift-machineconfig-master-dummy-networks
machineconfig.machineconfiguration.openshift.io/99-node-sizing-for-crc
machineconfig.machineconfiguration.openshift.io/99-worker-ssh
machineconfig.machineconfiguration.openshift.io/99-master-ssh
machineconfig.machineconfiguration.openshift.io/00-worker
machineconfig.machineconfiguration.openshift.io/01-master-container-runtime
machineconfig.machineconfiguration.openshift.io/01-master-kubelet
machineconfig.machineconfiguration.openshift.io/01-worker-container-runtime
machineconfig.machineconfiguration.openshift.io/01-worker-kubelet
machineconfig.machineconfiguration.openshift.io/99-master-generated-registries
machineconfig.machineconfiguration.openshift.io/00-master
machineconfig.machineconfiguration.openshift.io/99-worker-generated-registries
machineconfig.machineconfiguration.openshift.io/rendered-master-106dd41a9320b2f12717cabdb52fbc78
machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283
machineconfig.machineconfiguration.openshift.io/rendered-master-82f2facd6871b5ae714d735ad4a9a5b2
machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97
machineconfig.machineconfiguration.openshift.io/rendered-master-f67e28e8e1e4b1e223f544aa125c9d37
machineconfig.machineconfiguration.openshift.io/rendered-master-b2614334c19dc9f335083585765ea9d0
machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io/rendered-master-d0d5dc695ecc939aa643095cc77a2219
machineconfig.machineconfiguration.openshift.io/rendered-worker-f378fa4fa5c7bc150ac7925c3b4e4327'
+ grep rendered-worker
+ head -n -1
+ xargs -t ./openshift-clients/linux/oc delete
./openshift-clients/linux/oc delete machineconfig.machineconfiguration.openshift.io/rendered-worker-3559683dc13f9a32a1c7f82c1761a283 machineconfig.machineconfiguration.openshift.io/rendered-worker-7e0440ebe6381621e527a773a4f42d97 machineconfig.machineconfiguration.openshift.io/rendered-worker-32ca10bba66026336f26ca8794c3826d
machineconfig.machineconfiguration.openshift.io "rendered-worker-3559683dc13f9a32a1c7f82c1761a283" deleted
machineconfig.machineconfiguration.openshift.io "rendered-worker-7e0440ebe6381621e527a773a4f42d97" deleted
machineconfig.machineconfiguration.openshift.io "rendered-worker-32ca10bba66026336f26ca8794c3826d" deleted
+ jobs=($(jobs -p))
++ jobs -p
+ '[' -n '' ']'
+ true
Why are you trying 4.13.0.ec.2? latest
ppc64le build is 4.13.2 https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp/latest-4.13/
Why are you trying 4.13.0.ec.2? latest ppc64le build is 4.13.2 https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp/latest-4.13/
as that's the value we get by default snc.sh
from here - https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/latest-4.13/release.txt from here - https://github.com/crc-org/snc/blob/master/snc.sh#L34
check logs below .
export LC_ALL=C.UTF-8
+ LC_ALL=C.UTF-8
+ export LANG=C.UTF-8
+ LANG=C.UTF-8
+ source tools.sh
++ JQ=jq
++ QEMU_IMG=qemu-img
++ VIRT_FILESYSTEMS=virt-filesystems
++ GUESTFISH=guestfish
++ VIRSH=virsh
++ XMLLINT=xmllint
++ DIG=dig
++ UNZIP=unzip
++ ZSTD=zstd
++ CRC_ZSTD_EXTRA_FLAGS='--ultra -22'
++ HTPASSWD=htpasswd
++ PATCH=patch
+++ uname -m
++ ARCH=ppc64le
++ case "${ARCH}" in
++ yq_ARCH=ppc64le
++ SNC_GENERATE_MACOS_BUNDLE=0
++ SNC_GENERATE_WINDOWS_BUNDLE=0
++ SNC_GENERATE_LINUX_BUNDLE=1
++ test -z
++ echo 'Downloading yq binary to manipulate yaml files'
Downloading yq binary to manipulate yaml files
++ curl -L https://github.com/mikefarah/yq/releases/download/v4.5.1/yq_linux_ppc64le -o yq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 6599k 100 6599k 0 0 16.6M 0 --:--:-- --:--:-- --:--:-- 16.6M
++ chmod +x yq
++ YQ=./yq
++ which jq
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot jq
/usr/bin/jq
++ which virt-filesystems
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot virt-filesystems
++ alias
++ eval declare -f
+++ declare -f
/usr/bin/virt-filesystems
++ which guestfish
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot guestfish
/usr/bin/guestfish
++ which virsh
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot virsh
/usr/bin/virsh
++ which qemu-img
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot qemu-img
/usr/bin/qemu-img
++ rpm -q libguestfs-xfs
libguestfs-xfs-1.44.0-9.module+el8.8.0+16781+9f4724c2.ppc64le
++ '[' 0 '!=' 0 -o 0 '!=' 0 ']'
++ which xmllint
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot xmllint
/usr/bin/xmllint
++ which dig
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot dig
/usr/bin/dig
++ which zstd
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot zstd
/usr/bin/zstd
++ which htpasswd
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot htpasswd
/usr/bin/htpasswd
++ which patch
++ alias
++ eval declare -f
+++ declare -f
++ /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot patch
/usr/bin/patch
+ source snc-library.sh
++ set -exuo pipefail
+ trap 'jobs=($(jobs -p)); [ -n "${jobs-}" ] && ((${#jobs})) && kill "${jobs[@]}" || true' EXIT
+ OKD_VERSION=none
+ BUNDLE_TYPE=snc
+ [[ none != \n\o\n\e ]]
+ INSTALL_DIR=crc-tmp-install-data
+ SNC_PRODUCT_NAME=crc
+ SNC_CLUSTER_MEMORY=14336
+ SNC_CLUSTER_CPUS=6
+ CRC_VM_DISK_SIZE=33285996544
+ BASE_DOMAIN=testing
+ CRC_PV_DIR=/mnt/pv-data
+ SSH='ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ SCP='scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i id_ecdsa_crc'
+ MIRROR=https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview
+ CERT_ROTATION=enabled
+ USE_PATCHED_RELEASE_IMAGE=disabled
+ HTPASSWD_FILE=users.htpasswd
+ run_preflight_checks snc
+ local bundle_type=snc
+ '[' -z /home/amit_snc/pull_secret.json ']'
+ '[' '!' -f /home/amit_snc/pull_secret.json ']'
+ echo 'Checking libvirt and DNS configuration'
Checking libvirt and DNS configuration
+ LIBVIRT_URI=qemu:///system
+ '[' snc == snc ']'
+ LIBVIRT_URI=qemu+tcp://localhost/system
+ virsh -c qemu+tcp://localhost/system uri
+ virsh -c qemu+tcp://localhost/system net-info default
+ echo 'default libvirt network is available'
default libvirt network is available
++ virsh -c qemu+tcp://localhost/system net-info default
++ awk '{print $2}'
++ sed '3q;d'
+ [[ yes == \n\o ]]
+ case $ARCH in
+ echo 'The host arch is ppc64le.'
The host arch is ppc64le.
+ virsh -c qemu+tcp://localhost/system capabilities
+ xmllint --xpath '/capabilities/guest/arch[@name='\''ppc64le'\'']/domain[@type='\''kvm'\'']' -
+ '[' snc == snc ']'
+ local ping_status
++ ping -c1 api.crc.testing
++ head -1
+ ping_status='PING api.crc.testing (192.168.126.11) 56(84) bytes of data.'
+ grep 'PING api.crc.testing ('
+ echo PING api.crc.testing '(192.168.126.11)' '56(84)' bytes of data.
PING api.crc.testing (192.168.126.11) 56(84) bytes of data.
+ echo PING api.crc.testing '(192.168.126.11)' '56(84)' bytes of data.
+ grep '192.168.126.1[01])'
PING api.crc.testing (192.168.126.11) 56(84) bytes of data.
+ echo 'libvirt and DNS configuration successfully checked'
libvirt and DNS configuration successfully checked
+ test -n ''
++ curl -L https://mirror.openshift.com/pub/openshift-v4/ppc64le/clients/ocp-dev-preview/latest-4.13/release.txt
++ sed -n 's/^ *Version: *//p'
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 31548 100 31548 0 0 405k 0 --:--:-- --:--:-- --:--:-- 405k
+ OPENSHIFT_RELEASE_VERSION=4.13.0-ec.2
Using latest 4.13 via below change , thanks @cfergeau !!!
-MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp-dev-preview}
+MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp}
getting better results .
[root@ltc22briggs020 snc]# ./openshift-clients/linux/oc get clusterversion
NAME VERSION AVAILABLE PROGRESSING SINCE STATUS
version 4.13.2 True False 24h Cluster version is 4.13.2
[root@ltc22briggs020 snc]# ./openshift-clients/linux/oc get co
NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.13.2 True False False 33m
config-operator 4.13.2 True False False 25h
console 4.13.2 True False False 24m
control-plane-machine-set 4.13.2 True False False 25h
dns 4.13.2 True False False 39m
etcd 4.13.2 True False False 25h
image-registry 4.13.2 True False False 24h
ingress 4.13.2 True False False 25h
kube-apiserver 4.13.2 True False False 24h
kube-controller-manager 4.13.2 True False False 24h
kube-scheduler 4.13.2 True False False 24h
kube-storage-version-migrator 4.13.2 True False False 25h
machine-api 4.13.2 True False False 24h
machine-approver 4.13.2 True False False 25h
machine-config 4.13.2 True False False 25h
marketplace 4.13.2 True False False 25h
network 4.13.2 True False False 25h
openshift-apiserver 4.13.2 True False False 24m
openshift-controller-manager 4.13.2 True False False 34m
openshift-samples 4.13.2 True False False 24h
operator-lifecycle-manager 4.13.2 True False False 25h
operator-lifecycle-manager-catalog 4.13.2 True False False 25h
operator-lifecycle-manager-packageserver 4.13.2 True False False 29m
service-ca 4.13.2 True False False 25h
Using latest 4.13 via below change , thanks @cfergeau !!!
-MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp-dev-preview} +MIRROR=${MIRROR:-https://mirror.openshift.com/pub/openshift-v4/$ARCH/clients/ocp}
This was a bug in snc
, we forgot to update it, thanks for pointing it out! It's fixed in master now: https://github.com/crc-org/snc/commit/6575a3c552438be76e73c4feb2325752912c4002
@ghatwala should we close this issue now?
Yes , closing this issue !!!
Hi all,
Using this trying to install SNC and facing below issues on a power 9 server. Attaching installation logs for reference with
./snc.sh &> snc-cmdrunlog.txt
. snc-cmdrunlog.txt . Any pointers to resolve the issue ?Checking for logs..
checking failing pod logs
Another pod which is failing due to missing power arch support for the image.