search

adfinis / openshift-etcd-backup

CronJob to perform an etcd backup inside an OpenShift cluster
GNU Affero General Public License v3.0
16 stars 18 forks source link

Unable to execute CronJob using rook-ceph default settings #87

Open okdadmin opened 1 month ago

okdadmin commented 1 month ago

Hi,

I want to install this app on my okd 4.13 cluster!

oc get no NAME STATUS ROLES AGE VERSION master1.okddev. Ready control-plane,master 21d v1.26.9+636f2be master2.okddev. Ready control-plane,master 21d v1.26.9+636f2be master3.okddev. Ready control-plane,master 21d v1.26.9+636f2be worker1.okddev. Ready worker 20d v1.26.9+636f2be worker2.okddev. Ready worker 20d v1.26.9+636f2be worker3.okddev. Ready worker 20d v1.26.9+636f2be worker4.okddev. Ready worker 20d v1.26.9+636f2be

oc edit schedulers.config.openshift.io cluster spec: mastersSchedulable: false policy: name: "" status: {}

My problem: MountVolume.MountDevice failed for volume "pvc-787e821f-9338-429c-bf7a-3dd5581abac2" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name rook-ceph.cephfs.csi.ceph.com not found in the list of registered CSI drivers.

It is wrong because scheduling is disabled on the masters. I tried it even if it is enabled and it works. Any idea how to overcome this problem? Laci

tongpu commented 1 month ago

I have feeling your issue is that the cronjob needs to be scheduled to the master nodes, but the CSI integration is not configured to be available on the master nodes.

Could you please provide the output of the following commands:

okdadmin commented 1 month ago

oc get nodes output

apiVersion: v1
items:
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      k8s.ovn.org/host-addresses: '["10.248.153.13"]'
      k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","interface-id":"br-ex_master1.okddev.mav.hu","mac-address":"00:50:56:8d:d2:d1","ip-addresses":["10.248.153.13/24"],"ip-address":"10.248.153.13/24","next-hops":["10.248.153.254"],"next-hop":"10.248.153.254","node-port-enable":"true","vlan-id":"0"}}'
      k8s.ovn.org/node-chassis-id: 821d3e9e-afa9-42c0-ae83-7f4b9eed11c1
      k8s.ovn.org/node-gateway-router-lrp-ifaddr: '{"ipv4":"100.64.0.4/16"}'
      k8s.ovn.org/node-mgmt-port-mac-address: da:85:bb:01:36:e5
      k8s.ovn.org/node-primary-ifaddr: '{"ipv4":"10.248.153.13/24"}'
      k8s.ovn.org/node-subnets: '{"default":["10.130.0.0/23"]}'
      machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
      machineconfiguration.openshift.io/currentConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: "7988082"
      machineconfiguration.openshift.io/reason: ""
      machineconfiguration.openshift.io/state: Done
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2024-05-24T05:48:15Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: master1.okddev.mav.hu
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
      node.openshift.io/os_id: fedora
    name: master1.okddev.mav.hu
    resourceVersion: "10882232"
    uid: 76e66d18-aa80-4c45-b420-3d2341c74c3a
  spec:
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
  status:
    addresses:
    - address: 10.248.153.13
      type: InternalIP
    - address: master1.okddev.mav.hu
      type: Hostname
    allocatable:
      cpu: 3500m
      ephemeral-storage: "95018478229"
      hugepages-2Mi: "0"
      memory: 15211044Ki
      pods: "250"
    capacity:
      cpu: "4"
      ephemeral-storage: 104266732Ki
      hugepages-2Mi: "0"
      memory: 16362020Ki
      pods: "250"
    conditions:
    - lastHeartbeatTime: "2024-06-14T11:43:35Z"
      lastTransitionTime: "2024-05-24T11:08:53Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2024-06-14T11:43:35Z"
      lastTransitionTime: "2024-05-24T11:08:53Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2024-06-14T11:43:35Z"
      lastTransitionTime: "2024-05-24T11:08:53Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2024-06-14T11:43:35Z"
      lastTransitionTime: "2024-05-24T11:09:05Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - quay.io/openshift/okd-content@sha256:8c1eec3797870b6ef486f18c4426159182bd0b58d0aa0bdb93d6babea9e702aa
      - quay.io/openshift/okd-content@sha256:946b533b74f08bb9b31dcd328bf7f5ffee266233b1960e58818c74f77a3f8155
      sizeBytes: 1598344561
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:53f053b5b6b9f77e4bc71d7f8066841853227ab76be8c9cccb24c357caaf0d8b
      sizeBytes: 1428406446
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:89a30bd87a9602f76300f875204aa3cdc141fc4a661351b6721db2bd36559989
      sizeBytes: 1428393126
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:f15e3b99a11aa0b508d1bb692190fa877f413f8083e25e3570380eca820b4796
      - registry.access.redhat.com/redhat/community-operator-index@sha256:f78ec858360ea90359687a47424d1ffc486692005f2aaae4264097fed14378a5
      - registry.access.redhat.com/redhat/community-operator-index:v4.13
      sizeBytes: 1428380852
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:aca538a8c7d76978b04813c907d5f1e9389f85721fff3d9fa52cf6dcf5c40482
      sizeBytes: 1428334772
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:599df4074c2125c61ef5572ba9e5baff06e1df8c29b75ef7b95a891e0660a2c3
      sizeBytes: 1428223147
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:a0a5222c6a882867acf36427765530db0eb46cfb6f2f3ab7034f6bc892de89fc
      sizeBytes: 1417168044
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:99b05696d2d322fa63e963d505f7b4ae42ac499c3aba1a74c970e70feaaa1241
      sizeBytes: 1417093294
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:efd3824ab069b618c042e756586623d9553161d2937db523af26d0c4ddb8af32
      sizeBytes: 1417093289
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:452cf353e498ca0d267104a2904dd43c3e0745187ca28bd8da554a4cc7295e09
      sizeBytes: 1415566502
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ce96849c38970ae66034e544cbc9dd664ab6ad6db2ef33d453c6aa5604733358
      sizeBytes: 1415498417
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:0ce3ab955a84c62d3e08e112c9c93ef8e680c94986fd23a6ae8964fc267e78a0
      sizeBytes: 1415421099
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ae373df9e74de6980f19e0c6e66d8bca62d0a36c39641131c9212d4a1c785b6a
      sizeBytes: 1415402159
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:03d0bfc4e746ecbaff313e42fd3bd5ecb54a2d289f547d970de120fb4b2cb6e5
      sizeBytes: 1415070374
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:8bb2f9e6b79b181da6b56762fee85b26bafd343544cb57a031dbd739d5d9ee11
      sizeBytes: 1415035563
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:3827baa011f331e348657e9c7c707dea556eb52d2df1d0e38e27d53ae5e5ed5a
      sizeBytes: 1414868642
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:0e1462a02472244701b5ecbad201be8c086814d2b80bdb3e5c0dbbb46c01a0c8
      sizeBytes: 1414830756
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:40d05ff2afbd1c4d368118a495389d52debfd7111101d548c1fbd3c3d4de10bc
      sizeBytes: 1414782121
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:0baddfaf9489de0af7bfcfe458934688f88c19137f5663b25cb59ed9291957ce
      sizeBytes: 1414772393
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:3510b43e79a1a108703655856ec3e91871d82875cb2324538e492afe55de15ec
      sizeBytes: 1414772387
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ab5d538d81dd65098a2c22c6013578658d91b0e99c755391746d31ab49016529
      sizeBytes: 1414768815
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:4dc34d99eeb639e55b9dbfb63b6954bdc8b83451ab6c021dd4ae8163cc4f5159
      sizeBytes: 1414742709
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:975524d7a7139f42e938ac63abefbbe2561753e3a27e88c043443c8ba4607bbb
      sizeBytes: 1414704817
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:153b96065a2d2ffc91efe02869f3b17524c4857660c3f7cbbaf007ff8a4bfd1f
      sizeBytes: 1414698679
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:86b9f93540d6b4f2de8b139dbb8fc5a1c8975d8d3ce787d36949754871daeafa
      sizeBytes: 1414686362
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:f10fe4f7008788d1d5e93d432d5a7c3537ba3093051a441eb8927d72200273dd
      sizeBytes: 1414605994
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:18aba410220c13cb5a81e6a29057919beb04b9c87e49a771edf9626703a83fed
      sizeBytes: 1414596765
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ebef196717a7b47d89447a39c3acf95daf5bd916bd2b435d5a9812975760a796
      sizeBytes: 1414427316
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:7c71fc5fbacdaa848795844649e8815c50274c33c36c1a2735f6316b3817d6e9
      sizeBytes: 1414424233
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:b7f0aa1e6bc4401e1eda5dbc1d893129ee2a8181a82228f67947d5cded40fedc
      sizeBytes: 1414410938
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:e12fd6bf7e5cfbdcfcbdc77ad1898d978435d03d4e28d4fb63a81621f91c05b1
      sizeBytes: 1414399660
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ab3d69d84ac038457dab8a27c49bac0748279901b7afbe873ea041ff0f0e797a
      sizeBytes: 1414368937
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:28a4162e8bb75e0a39f6c7a8a1f5680cd5200e05d78f9f8110096f7acff9e648
      sizeBytes: 1414332070
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:589ad46bf6d8d1c97fcaeb3f46e33d33d014888bcbbbbd0c0436d8a9eeb4d01b
      sizeBytes: 1414301352
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:95ad49aa8a1e08f8eaf9e72a46ee54b33674c5dba980e7f0c0d594b1802bfd7e
      sizeBytes: 1414287024
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ba4e64eebb42d36c719f07ff16d9b389a02230800b23627ad59db7a82c4fff91
      sizeBytes: 1414282924
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:1b517257b05c0d721746853bf9c1e2b302db65a6c217cbaf29748d5962486940
      sizeBytes: 1414225559
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:435b0e5840b1c6fb74f501c49ae7c54a2129867ca45ccba1be029d4cdfcf81af
      sizeBytes: 1414150829
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:6b6beb74ace2ec008b0ac13c0bedb0c3274ea091111a1cef71b6855f8cbaa2fd
      sizeBytes: 1414027418
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:bbaf202f4b799a47cae447d395ee4bdc5a6f72e216678301660a18502ede1499
      sizeBytes: 1414017710
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ce10c1fd9f9aa719c6f383032d2ed40ff2b019e72a30d2a007a45139547f5440
      sizeBytes: 1414004381
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:97b1c51f8f98d7a97e28f0b5f7bac996eb41416f17b1e45332b2c4cd6b2888cc
      sizeBytes: 1413982882
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:d8f38b5e9c07c5e03c80627cfc191c0b56e6e88be0db86c9e7c6067baf00017a
      sizeBytes: 1413928107
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:91f207656302f5804c843f4cad1e9bdeba4f195ebef3909e18b0fdd1336ce617
      sizeBytes: 1413925544
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:999c8eb366f63f0faec2a95b684da4cf9430d40d78ef43c70daafebec9a6aa21
      sizeBytes: 1413916846
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:ec57461e63cf6b91a1cc2a0b0c1b82b50351233734637671ffc158685db09d95
      sizeBytes: 1413905050
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:54cf360a4cddeca3ccd6429bea9c20d87402f391dc70d963b05d190aa0d15481
      sizeBytes: 1413896358
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:19e6d649badb59b843d802a04ce26cef0d097f9d7090150601be1d1b3eef24d0
      sizeBytes: 1413893807
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:aa9566d5260e0f83d18dba446916d731cea9e36886751249666863beccf06613
      sizeBytes: 1413888679
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:8571aea6d89552ae54955b27c58ce38054580fdf52090628fe94088b5f147fa8
      sizeBytes: 1413851805
    nodeInfo:
      architecture: amd64
      bootID: 3052bbc9-6380-49e3-8903-dbe3d59654b7
      containerRuntimeVersion: cri-o://1.26.4
      kernelVersion: 6.5.5-200.fc38.x86_64
      kubeProxyVersion: v1.26.9+636f2be
      kubeletVersion: v1.26.9+636f2be
      machineID: f5dce322adb74f5482b0f8582c278370
      operatingSystem: linux
      osImage: Fedora CoreOS 38.20231002.3.1
      systemUUID: 1ce30d42-23a7-6f2d-a157-9693a781d3fa
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      k8s.ovn.org/host-addresses: '["10.248.153.14"]'
      k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","interface-id":"br-ex_master2.okddev.mav.hu","mac-address":"00:50:56:8d:93:a3","ip-addresses":["10.248.153.14/24"],"ip-address":"10.248.153.14/24","next-hops":["10.248.153.254"],"next-hop":"10.248.153.254","node-port-enable":"true","vlan-id":"0"}}'
      k8s.ovn.org/node-chassis-id: 1bdab7d6-baa0-449a-bb35-25a5b8c1e93e
      k8s.ovn.org/node-gateway-router-lrp-ifaddr: '{"ipv4":"100.64.0.2/16"}'
      k8s.ovn.org/node-mgmt-port-mac-address: 26:f8:7e:ca:ce:b0
      k8s.ovn.org/node-primary-ifaddr: '{"ipv4":"10.248.153.14/24"}'
      k8s.ovn.org/node-subnets: '{"default":["10.128.0.0/23"]}'
      machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
      machineconfiguration.openshift.io/currentConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: "7988082"
      machineconfiguration.openshift.io/reason: ""
      machineconfiguration.openshift.io/state: Done
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2024-05-24T05:37:26Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: master2.okddev.mav.hu
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
      node.openshift.io/os_id: fedora
    name: master2.okddev.mav.hu
    resourceVersion: "10881237"
    uid: 6bf7f857-6701-4d6a-a9a8-13f3cc791834
  spec:
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
  status:
    addresses:
    - address: 10.248.153.14
      type: InternalIP
    - address: master2.okddev.mav.hu
      type: Hostname
    allocatable:
      cpu: 3500m
      ephemeral-storage: "95018478229"
      hugepages-2Mi: "0"
      memory: 15211044Ki
      pods: "250"
    capacity:
      cpu: "4"
      ephemeral-storage: 104266732Ki
      hugepages-2Mi: "0"
      memory: 16362020Ki
      pods: "250"
    conditions:
    - lastHeartbeatTime: "2024-06-14T11:40:46Z"
      lastTransitionTime: "2024-05-24T10:58:20Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2024-06-14T11:40:46Z"
      lastTransitionTime: "2024-05-24T10:58:20Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2024-06-14T11:40:46Z"
      lastTransitionTime: "2024-05-24T10:58:20Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2024-06-14T11:40:46Z"
      lastTransitionTime: "2024-05-24T10:58:21Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - quay.io/openshift/okd-content@sha256:8c1eec3797870b6ef486f18c4426159182bd0b58d0aa0bdb93d6babea9e702aa
      - quay.io/openshift/okd-content@sha256:946b533b74f08bb9b31dcd328bf7f5ffee266233b1960e58818c74f77a3f8155
      sizeBytes: 1598344561
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:738e57333d273de2d92604591b0be5173d8b957cfe2a85460e4240d0eb694ee7
      sizeBytes: 1414184092
    - names:
      - registry.access.redhat.com/redhat/community-operator-index@sha256:1cd0d0ca521cacd84cc7b47ab92bde4a438e74908f95d03816599d9f851cc3a5
      - registry.access.redhat.com/redhat/community-operator-index@sha256:81d563fff9874f6ccff8da86281269280b2dad42b160096c4745c9b401d47a0d
      - registry.access.redhat.com/redhat/community-operator-index:v4.13
      sizeBytes: 1413026977
    - names:
      - quay.io/openshift/okd-content@sha256:26786007ba238a6ecd2d16318a56628e370407fffc58d104a4162d5704ce0888
      - quay.io/openshift/okd-content@sha256:fcc5e9dc0e3e62f048f495a54fea56cd952abc8fac97ca65a7750641770a1cf0
      sizeBytes: 1132309475
    - names:
      - quay.io/openshift/okd-content@sha256:24ee7cb9664aad8d87d9edcf206bbab72d6cd661ca3f3fdbaebd47215d770bf1
      - quay.io/openshift/okd-content@sha256:8821e13ec1a2d4882a79bb1e79bfc45fe52b4f2f8bd4beea78e6aa64ea6119b9
      sizeBytes: 934367239
    - names:
      - quay.io/openshift/okd-content@sha256:d8a6c758b2d88f3cc7af5de179f7af0c791160e134a583dbe929b5ba733a8329
      sizeBytes: 805394159
    - names:
      - quay.io/openshift/okd-content@sha256:71f069300bd9e99ed8ddb3731dddb7660659259c752e2bd30ef695e5df932dbd
      - quay.io/openshift/okd-content@sha256:db5fbe60526bc2b969d9770116a6bc272327c10cbeb0dc98cbc0f229067ad33c
      sizeBytes: 796084302
    - names:
      - quay.io/openshift/okd-content@sha256:96f167e4c97e0d36133a519e41a3c9d469d3d476ff62a6070327d008026f20b6
      - quay.io/openshift/okd-content@sha256:f6f828b07146ee6a9b800fbd5f1fac9cbba49d38beb3c2a04f1594b9473cd101
      sizeBytes: 738568542
    - names:
      - quay.io/openshift/okd-content@sha256:a9f473ffab89cc8227a12b0174248ef31cf5c951aaa1b1f9188a5ce1ab79f199
      sizeBytes: 687356700
    - names:
      - quay.io/openshift/okd-content@sha256:e310e8163ce7bebc58da8c38c28e4e475e2ab08bbb26c9012fa77badea9db5cd
      sizeBytes: 682475084
    - names:
      - quay.io/openshift/okd-content@sha256:e6e8368d2b86ef93e9772fa1f3396a2a04b4ade9f63d4c3b054465e84247b578
      sizeBytes: 673870719
    - names:
      - quay.io/openshift/okd-content@sha256:634707d1a600681550ff317bea333f767c5e83180e7770798c3505844b171184
      sizeBytes: 654736240
    - names:
      - quay.io/openshift/okd-content@sha256:43bf86a8dfd5684edb2eed466b157f9a93451d83afc04aef98a3769a4691856f
      - quay.io/openshift/okd-content@sha256:496a082ea769759959114fe36d2ad4443be0b92dfd261183df284fffbb8b0d66
      sizeBytes: 629094110
    - names:
      - quay.io/openshift/okd-content@sha256:0d5be1c7655d50da9966e48a6063174291c07c09252f01efd15134a1691de290
      - quay.io/openshift/okd-content@sha256:f2dce33759b4d3cc8ed8bf604e44f23265bce990f229a32acb40c87b5c6ae72c
      sizeBytes: 612768474
    - names:
      - quay.io/openshift/okd-content@sha256:0be351bdf1d1697aaacfec98e9a4d029bae8921e68f895ed669f805cd4c46680
      - quay.io/openshift/okd-content@sha256:9bcd46b82c636157cf840200968a0a446e88e95055cb3abb353062cc28145e48
      sizeBytes: 593445683
    - names:
      - quay.io/openshift/okd-content@sha256:55a2be37a97f4d26f50692eec1e0abac7ac70549517826966671a094aa09f8ba
      - quay.io/openshift/okd-content@sha256:b3433c1da485162602f6430d163d792f77f82b04bbbefbc01a20bd23443f009c
      sizeBytes: 567491465
    - names:
      - quay.io/openshift/okd-content@sha256:3a1eabed58de366b83daf5170d70004a81bdf6175fd48280d35ab8e985fc35db
      - quay.io/openshift/okd-content@sha256:9a8f12c03af80911d16812f56e298f9c2f99615086791c744115c34bdcb616bd
      sizeBytes: 543940053
    - names:
      - quay.io/openshift/okd-content@sha256:cee191136c2d2006db7f72fd6ee95460c7fc0b372b6f66ab3c6435469fb5c850
      - quay.io/openshift/okd-content@sha256:de0a331fb5dc22642b6ba67d5777b5b80baf1cac349151e08479f45703814178
      sizeBytes: 540694863
    - names:
      - quay.io/openshift/okd-content@sha256:d4aa445ede3f1f72d30fbd58e902e75ad4ca12fa34f7e88cdbe1e285b627a928
      - quay.io/openshift/okd-content@sha256:eb75e46a2ebc28a72176b1ad08c79cd0bb72a796fc9a5d064d604f60c32e3f44
      sizeBytes: 540684142
    - names:
      - quay.io/openshift/okd-content@sha256:3c0853dc9476407408319ce2e311382589df43e75112a58f7c61cdfa8afbc6f6
      - quay.io/openshift/okd-content@sha256:66fde9ae7a5dd84bf326ee36aee50714da6475ba64bbe548141f711c3bddde31
      sizeBytes: 539079391
    - names:
      - quay.io/openshift/okd-content@sha256:ae840b5328bb94129128f15063422a271e71139639f29a7d36a0808dfbe618b4
      - quay.io/openshift/okd-content@sha256:b7e63c1db5a850439a536216468205662f25f123241d171c28923728de2b4c57
      sizeBytes: 534769830
    - names:
      - quay.io/openshift/okd-content@sha256:35248da422b11c2afa85c0b737c7c9814473c7c9785571cb6b19c336011cb1bb
      - quay.io/openshift/okd-content@sha256:53306d7ccc280fc18548b2cd7bda6684e20e83b1f6a4bb7ad111abac9615a5c3
      sizeBytes: 533657528
    - names:
      - quay.io/openshift/okd-content@sha256:64245b1ea4104385ffd4a4a2a776f92b736a2404738fb212ea3d1a9a734ad895
      - quay.io/openshift/okd-content@sha256:826196f0ddcac7f6a21672f268c23e1f850e2bc3cb85d9da25aff0c8aa3af293
      sizeBytes: 533143390
    - names:
      - quay.io/openshift/okd@sha256:29e8ad18b5d011c3fd8275ec9aad5964e4dcfd0138d229f2917acfcbefde930c
      sizeBytes: 524903462
    - names:
      - quay.io/openshift/okd-content@sha256:14452482c6b2ce51aac6f915ea4e06636a193e3568ee1520563f745a69aeec3d
      - quay.io/openshift/okd-content@sha256:2d3bc5f8d8e488c6f81a922508bab56033b1ad2c61423bf1254f6d5a33b180e2
      sizeBytes: 519038548
    - names:
      - quay.io/openshift/okd-content@sha256:4b12180d8a74e8c27eb1380bc83e8104471410f354f7bb9effbb115268706ad8
      sizeBytes: 511418976
    - names:
      - quay.io/openshift/okd-content@sha256:3c9d6c190686d49f7c5e45dfc653d8ad10fd882c8be987c28b5b5a6ff0c4538d
      sizeBytes: 511124338
    - names:
      - quay.io/openshift/okd-content@sha256:11de35c0696258b1e4ed055b9c6e80dbb5924dada1bfe460292291a28ffbfa0f
      - quay.io/openshift/okd-content@sha256:cde74f8f262a7c040626d078f9abc0358eef7dc83db92c0ce90f808b1f6ad20f
      sizeBytes: 509150399
    - names:
      - quay.io/openshift/okd-content@sha256:e9b7596667d754836bd2a7bd3a5dac4a80fe7f4fb520aba281e23acff3ae6a6a
      - quay.io/openshift/okd-content@sha256:fb4f95c4ceac39ca295e400ea5447cbc31e662c49e884b04325aba9c482cfed0
      sizeBytes: 507900943
    - names:
      - quay.io/openshift/okd-content@sha256:d142ff7f0c8447f9bf1b316b3a86bc26d62db620989e393470980e8e3de4ec6f
      sizeBytes: 500632931
    - names:
      - quay.io/openshift/okd-content@sha256:436a9927d32e8aee5aed835733b61890302e7f20cb34ed68d6c31e866ab41a0f
      sizeBytes: 498976823
    - names:
      - quay.io/openshift/okd-content@sha256:0d9ce55695ec9c655b7ce8b1ff539fbee08ad2dce4b645c289ca6f12e596f195
      - quay.io/openshift/okd-content@sha256:9453238772013e5c086019a12e1212d9c4479b98aa30015a61aeadae4d2619fd
      sizeBytes: 498798132
    - names:
      - quay.io/openshift/okd-content@sha256:5dd2972b1f7ab80b5d64dd629757030e3736b863ae47f7b9a8d5be120c146c5e
      - quay.io/openshift/okd-content@sha256:7ee7df898db7e8191b8ee9130959ef73f0db4b4630160d994d576026e36dfd6c
      sizeBytes: 498093706
    - names:
      - quay.io/openshift/okd-content@sha256:74e236411df9f3c2907bf5d648362ddf9fc925f9851c44544b1a8f64f498311b
      - quay.io/openshift/okd-content@sha256:c3a5a41c857796f1bcfdec8f9e85c31911901d8eaff63e081d9ea81885c7f430
      sizeBytes: 497324751
    - names:
      - quay.io/openshift/okd-content@sha256:048dea4554d7cc6dfaed90579278d66df99b643805b0ec018206ebd8f35f4e4b
      sizeBytes: 496302234
    - names:
      - quay.io/openshift/okd-content@sha256:8607cba601536c65a826ccccb423b8dbd9574e2ec3b9536bda75150b049b3eb7
      - quay.io/openshift/okd-content@sha256:b5ae1edc64df1fa875f9deeebd0af68adfe98f3535c2b6b1aac38e6077b25d34
      sizeBytes: 496022037
    - names:
      - quay.io/openshift/okd-content@sha256:33da219fc16b3c2147dfbc5fdf5218cf28d6b70fcf16dd3af5d923958144dd0d
      - quay.io/openshift/okd-content@sha256:8dbeaa159ba785b37adb3070c7c8d7ea184b1ec346065686f3ac6c0007dae5c3
      sizeBytes: 495670212
    - names:
      - quay.io/openshift/okd-content@sha256:8dc04fcaaf860f4d7ab7d1d0f807256b91da030cab293447b639a2264c0747df
      sizeBytes: 495140160
    - names:
      - quay.io/openshift/okd-content@sha256:0159217f85c3402013e04be28cc4156ba48c12f92a70616ec89f3e9895b20254
      sizeBytes: 494654430
    - names:
      - quay.io/openshift/okd-content@sha256:ce71a63c3f4419b77e72bf8928a06a430b50cbf141bec8a7ff8dbd38bc81b69c
      sizeBytes: 492381581
    - names:
      - quay.io/openshift/okd-content@sha256:8afdec954ef25fea78f69fb94a81254c377edddb968a5a730651e723189b7dfa
      - quay.io/openshift/okd-content@sha256:c637e6cc52bf132c2d23af5a993d213ce453306aaa7ff473840d5f66756d5cc7
      sizeBytes: 489474154
    - names:
      - quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6
      sizeBytes: 483672938
    - names:
      - quay.io/openshift/okd-content@sha256:5a79d38a9613dc33b27cd2775d35613c0694c461b97135a04ebdbb7307532f47
      sizeBytes: 479923890
    - names:
      - quay.io/openshift/okd-content@sha256:fc90d9259e8f66cee8bd4e8ca187675331b2ef998e880cd04a0367e09ea8d035
      sizeBytes: 479827515
    - names:
      - quay.io/openshift/okd-content@sha256:3beac2d94e3006ace2cc937b691da6aa22ce0dba5c35d00bb497a7a5cd73068a
      - quay.io/openshift/okd-content@sha256:dd286a23ab14a2f262b67a76f23414a36bfbd7466f8ae94ba59dff659d3e270f
      sizeBytes: 478268055
    - names:
      - quay.io/openshift/okd-content@sha256:25d10ff56512914765c6d9f340d9ad6807cd10de9b804218de74a64f8be4b257
      sizeBytes: 472976545
    - names:
      - quay.io/openshift/okd-content@sha256:c8307b4291975183adc6a835e621e8c74d729ba611caa09c986f799104d24fc4
      sizeBytes: 461768877
    - names:
      - quay.io/openshift/okd-content@sha256:3392b9127b29b50e578618a36ce6ec3f3ae741337db5cfbb5618d36d1c235100
      - quay.io/openshift/okd-content@sha256:8e94d708c98dadc984806f21837b4a71c2b613e4965f0782c363c33387e70467
      sizeBytes: 461074249
    - names:
      - quay.io/openshift/okd-content@sha256:2cbc4bf38c29285c2b8e4052e90e77720b98c366046be96c1382867c3f4287b1
      sizeBytes: 455911503
    - names:
      - quay.io/openshift/okd-content@sha256:74499ce45829545bd83876e76a5266b1db64e256d9a68d7673f4fb93aef0a272
      - quay.io/openshift/okd-content@sha256:8cd4b7925c31290234220e019c33db0137ab0e45701d975ee8c2bd723b95f722
      sizeBytes: 446650003
    nodeInfo:
      architecture: amd64
      bootID: 61ac0db6-1a66-4192-87be-37b12484520e
      containerRuntimeVersion: cri-o://1.26.4
      kernelVersion: 6.5.5-200.fc38.x86_64
      kubeProxyVersion: v1.26.9+636f2be
      kubeletVersion: v1.26.9+636f2be
      machineID: 5d669605fbed456d93e3998cb67d4b5f
      operatingSystem: linux
      osImage: Fedora CoreOS 38.20231002.3.1
      systemUUID: 89f00d42-d36a-02a8-f975-6859b1ab8587
- apiVersion: v1
  kind: Node
  metadata:
    annotations:
      k8s.ovn.org/host-addresses: '["10.248.153.15"]'
      k8s.ovn.org/l3-gateway-config: '{"default":{"mode":"shared","interface-id":"br-ex_master3.okddev.mav.hu","mac-address":"00:50:56:8d:59:44","ip-addresses":["10.248.153.15/24"],"ip-address":"10.248.153.15/24","next-hops":["10.248.153.254"],"next-hop":"10.248.153.254","node-port-enable":"true","vlan-id":"0"}}'
      k8s.ovn.org/node-chassis-id: 13689340-9707-4564-ae80-4ce826f7ea0b
      k8s.ovn.org/node-gateway-router-lrp-ifaddr: '{"ipv4":"100.64.0.3/16"}'
      k8s.ovn.org/node-mgmt-port-mac-address: 7e:a1:bb:61:0d:bd
      k8s.ovn.org/node-primary-ifaddr: '{"ipv4":"10.248.153.15/24"}'
      k8s.ovn.org/node-subnets: '{"default":["10.129.0.0/23"]}'
      machineconfiguration.openshift.io/controlPlaneTopology: HighlyAvailable
      machineconfiguration.openshift.io/currentConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredConfig: rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/desiredDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastAppliedDrain: uncordon-rendered-master-85663ccd098e296084d3f4eb83b073f2
      machineconfiguration.openshift.io/lastSyncedControllerConfigResourceVersion: "7988082"
      machineconfiguration.openshift.io/reason: ""
      machineconfiguration.openshift.io/state: Done
      volumes.kubernetes.io/controller-managed-attach-detach: "true"
    creationTimestamp: "2024-05-24T05:39:07Z"
    labels:
      beta.kubernetes.io/arch: amd64
      beta.kubernetes.io/os: linux
      kubernetes.io/arch: amd64
      kubernetes.io/hostname: master3.okddev.mav.hu
      kubernetes.io/os: linux
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
      node.openshift.io/os_id: fedora
    name: master3.okddev.mav.hu
    resourceVersion: "10882447"
    uid: 1b0d6f40-3340-4dbb-b6f1-f4a0cf79e983
  spec:
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
  status:
    addresses:
    - address: 10.248.153.15
      type: InternalIP
    - address: master3.okddev.mav.hu
      type: Hostname
    allocatable:
      cpu: 3500m
      ephemeral-storage: "95018478229"
      hugepages-2Mi: "0"
      memory: 15211044Ki
      pods: "250"
    capacity:
      cpu: "4"
      ephemeral-storage: 104266732Ki
      hugepages-2Mi: "0"
      memory: 16362020Ki
      pods: "250"
    conditions:
    - lastHeartbeatTime: "2024-06-14T11:44:11Z"
      lastTransitionTime: "2024-06-11T20:47:57Z"
      message: kubelet has sufficient memory available
      reason: KubeletHasSufficientMemory
      status: "False"
      type: MemoryPressure
    - lastHeartbeatTime: "2024-06-14T11:44:11Z"
      lastTransitionTime: "2024-06-11T20:47:57Z"
      message: kubelet has no disk pressure
      reason: KubeletHasNoDiskPressure
      status: "False"
      type: DiskPressure
    - lastHeartbeatTime: "2024-06-14T11:44:11Z"
      lastTransitionTime: "2024-06-11T20:47:57Z"
      message: kubelet has sufficient PID available
      reason: KubeletHasSufficientPID
      status: "False"
      type: PIDPressure
    - lastHeartbeatTime: "2024-06-14T11:44:11Z"
      lastTransitionTime: "2024-06-11T20:47:57Z"
      message: kubelet is posting ready status
      reason: KubeletReady
      status: "True"
      type: Ready
    daemonEndpoints:
      kubeletEndpoint:
        Port: 10250
    images:
    - names:
      - quay.io/openshift/okd-content@sha256:26786007ba238a6ecd2d16318a56628e370407fffc58d104a4162d5704ce0888
      - quay.io/openshift/okd-content@sha256:fcc5e9dc0e3e62f048f495a54fea56cd952abc8fac97ca65a7750641770a1cf0
      sizeBytes: 1132309475
    - names:
      - quay.io/openshift/okd-content@sha256:24ee7cb9664aad8d87d9edcf206bbab72d6cd661ca3f3fdbaebd47215d770bf1
      - quay.io/openshift/okd-content@sha256:8821e13ec1a2d4882a79bb1e79bfc45fe52b4f2f8bd4beea78e6aa64ea6119b9
      sizeBytes: 934367239
    - names:
      - quay.io/openshift/okd-content@sha256:d8a6c758b2d88f3cc7af5de179f7af0c791160e134a583dbe929b5ba733a8329
      sizeBytes: 805394159
    - names:
      - quay.io/openshift/okd-content@sha256:71f069300bd9e99ed8ddb3731dddb7660659259c752e2bd30ef695e5df932dbd
      - quay.io/openshift/okd-content@sha256:db5fbe60526bc2b969d9770116a6bc272327c10cbeb0dc98cbc0f229067ad33c
      sizeBytes: 796084302
    - names:
      - quay.io/openshift/okd-content@sha256:96f167e4c97e0d36133a519e41a3c9d469d3d476ff62a6070327d008026f20b6
      - quay.io/openshift/okd-content@sha256:f6f828b07146ee6a9b800fbd5f1fac9cbba49d38beb3c2a04f1594b9473cd101
      sizeBytes: 738568542
    - names:
      - quay.io/openshift/okd-content@sha256:a9f473ffab89cc8227a12b0174248ef31cf5c951aaa1b1f9188a5ce1ab79f199
      sizeBytes: 687356700
    - names:
      - quay.io/openshift/okd-content@sha256:e310e8163ce7bebc58da8c38c28e4e475e2ab08bbb26c9012fa77badea9db5cd
      sizeBytes: 682475084
    - names:
      - quay.io/openshift/okd-content@sha256:e6e8368d2b86ef93e9772fa1f3396a2a04b4ade9f63d4c3b054465e84247b578
      sizeBytes: 673870719
    - names:
      - quay.io/openshift/okd-content@sha256:634707d1a600681550ff317bea333f767c5e83180e7770798c3505844b171184
      sizeBytes: 654736240
    - names:
      - quay.io/openshift/okd-content@sha256:43bf86a8dfd5684edb2eed466b157f9a93451d83afc04aef98a3769a4691856f
      - quay.io/openshift/okd-content@sha256:496a082ea769759959114fe36d2ad4443be0b92dfd261183df284fffbb8b0d66
      sizeBytes: 629094110
    - names:
      - quay.io/openshift/okd-content@sha256:0d5be1c7655d50da9966e48a6063174291c07c09252f01efd15134a1691de290
      - quay.io/openshift/okd-content@sha256:f2dce33759b4d3cc8ed8bf604e44f23265bce990f229a32acb40c87b5c6ae72c
      sizeBytes: 612768474
    - names:
      - quay.io/openshift/okd-content@sha256:0be351bdf1d1697aaacfec98e9a4d029bae8921e68f895ed669f805cd4c46680
      - quay.io/openshift/okd-content@sha256:9bcd46b82c636157cf840200968a0a446e88e95055cb3abb353062cc28145e48
      sizeBytes: 593445683
    - names:
      - quay.io/openshift/okd-content@sha256:55a2be37a97f4d26f50692eec1e0abac7ac70549517826966671a094aa09f8ba
      - quay.io/openshift/okd-content@sha256:b3433c1da485162602f6430d163d792f77f82b04bbbefbc01a20bd23443f009c
      sizeBytes: 567491465
    - names:
      - quay.io/openshift/okd-content@sha256:3a1eabed58de366b83daf5170d70004a81bdf6175fd48280d35ab8e985fc35db
      - quay.io/openshift/okd-content@sha256:9a8f12c03af80911d16812f56e298f9c2f99615086791c744115c34bdcb616bd
      sizeBytes: 543940053
    - names:
      - quay.io/openshift/okd-content@sha256:cee191136c2d2006db7f72fd6ee95460c7fc0b372b6f66ab3c6435469fb5c850
      - quay.io/openshift/okd-content@sha256:de0a331fb5dc22642b6ba67d5777b5b80baf1cac349151e08479f45703814178
      sizeBytes: 540694863
    - names:
      - quay.io/openshift/okd-content@sha256:d4aa445ede3f1f72d30fbd58e902e75ad4ca12fa34f7e88cdbe1e285b627a928
      - quay.io/openshift/okd-content@sha256:eb75e46a2ebc28a72176b1ad08c79cd0bb72a796fc9a5d064d604f60c32e3f44
      sizeBytes: 540684142
    - names:
      - quay.io/openshift/okd-content@sha256:3c0853dc9476407408319ce2e311382589df43e75112a58f7c61cdfa8afbc6f6
      - quay.io/openshift/okd-content@sha256:66fde9ae7a5dd84bf326ee36aee50714da6475ba64bbe548141f711c3bddde31
      sizeBytes: 539079391
    - names:
      - quay.io/openshift/okd-content@sha256:ae840b5328bb94129128f15063422a271e71139639f29a7d36a0808dfbe618b4
      - quay.io/openshift/okd-content@sha256:b7e63c1db5a850439a536216468205662f25f123241d171c28923728de2b4c57
      sizeBytes: 534769830
    - names:
      - quay.io/openshift/okd-content@sha256:35248da422b11c2afa85c0b737c7c9814473c7c9785571cb6b19c336011cb1bb
      - quay.io/openshift/okd-content@sha256:53306d7ccc280fc18548b2cd7bda6684e20e83b1f6a4bb7ad111abac9615a5c3
      sizeBytes: 533657528
    - names:
      - quay.io/openshift/okd-content@sha256:64245b1ea4104385ffd4a4a2a776f92b736a2404738fb212ea3d1a9a734ad895
      - quay.io/openshift/okd-content@sha256:826196f0ddcac7f6a21672f268c23e1f850e2bc3cb85d9da25aff0c8aa3af293
      sizeBytes: 533143390
    - names:
      - quay.io/openshift/okd@sha256:29e8ad18b5d011c3fd8275ec9aad5964e4dcfd0138d229f2917acfcbefde930c
      sizeBytes: 524903462
    - names:
      - quay.io/openshift/okd-content@sha256:14452482c6b2ce51aac6f915ea4e06636a193e3568ee1520563f745a69aeec3d
      - quay.io/openshift/okd-content@sha256:2d3bc5f8d8e488c6f81a922508bab56033b1ad2c61423bf1254f6d5a33b180e2
      sizeBytes: 519038548
    - names:
      - quay.io/openshift/okd-content@sha256:4b12180d8a74e8c27eb1380bc83e8104471410f354f7bb9effbb115268706ad8
      sizeBytes: 511418976
    - names:
      - quay.io/openshift/okd-content@sha256:d14fae69d4457cd6f5e237af1aeb0e5520564ae0c90fbba5897d09ced28c3982
      - quay.io/openshift/okd-content@sha256:d8e1c33d6122163e84d05462f5342176cd7d5c27cfcb69b83011999fd88a57f4
      sizeBytes: 509976527
    - names:
      - quay.io/openshift/okd-content@sha256:11de35c0696258b1e4ed055b9c6e80dbb5924dada1bfe460292291a28ffbfa0f
      - quay.io/openshift/okd-content@sha256:cde74f8f262a7c040626d078f9abc0358eef7dc83db92c0ce90f808b1f6ad20f
      sizeBytes: 509150399
    - names:
      - quay.io/openshift/okd-content@sha256:e9b7596667d754836bd2a7bd3a5dac4a80fe7f4fb520aba281e23acff3ae6a6a
      - quay.io/openshift/okd-content@sha256:fb4f95c4ceac39ca295e400ea5447cbc31e662c49e884b04325aba9c482cfed0
      sizeBytes: 507900943
    - names:
      - quay.io/openshift/okd-content@sha256:d142ff7f0c8447f9bf1b316b3a86bc26d62db620989e393470980e8e3de4ec6f
      sizeBytes: 500632931
    - names:
      - quay.io/openshift/okd-content@sha256:436a9927d32e8aee5aed835733b61890302e7f20cb34ed68d6c31e866ab41a0f
      sizeBytes: 498976823
    - names:
      - quay.io/openshift/okd-content@sha256:0d9ce55695ec9c655b7ce8b1ff539fbee08ad2dce4b645c289ca6f12e596f195
      - quay.io/openshift/okd-content@sha256:9453238772013e5c086019a12e1212d9c4479b98aa30015a61aeadae4d2619fd
      sizeBytes: 498798132
    - names:
      - quay.io/openshift/okd-content@sha256:5dd2972b1f7ab80b5d64dd629757030e3736b863ae47f7b9a8d5be120c146c5e
      - quay.io/openshift/okd-content@sha256:7ee7df898db7e8191b8ee9130959ef73f0db4b4630160d994d576026e36dfd6c
      sizeBytes: 498093706
    - names:
      - quay.io/openshift/okd-content@sha256:74e236411df9f3c2907bf5d648362ddf9fc925f9851c44544b1a8f64f498311b
      - quay.io/openshift/okd-content@sha256:c3a5a41c857796f1bcfdec8f9e85c31911901d8eaff63e081d9ea81885c7f430
      sizeBytes: 497324751
    - names:
      - quay.io/openshift/okd-content@sha256:048dea4554d7cc6dfaed90579278d66df99b643805b0ec018206ebd8f35f4e4b
      sizeBytes: 496302234
    - names:
      - quay.io/openshift/okd-content@sha256:8607cba601536c65a826ccccb423b8dbd9574e2ec3b9536bda75150b049b3eb7
      - quay.io/openshift/okd-content@sha256:b5ae1edc64df1fa875f9deeebd0af68adfe98f3535c2b6b1aac38e6077b25d34
      sizeBytes: 496022037
    - names:
      - quay.io/openshift/okd-content@sha256:33da219fc16b3c2147dfbc5fdf5218cf28d6b70fcf16dd3af5d923958144dd0d
      - quay.io/openshift/okd-content@sha256:8dbeaa159ba785b37adb3070c7c8d7ea184b1ec346065686f3ac6c0007dae5c3
      sizeBytes: 495670212
    - names:
      - quay.io/openshift/okd-content@sha256:8dc04fcaaf860f4d7ab7d1d0f807256b91da030cab293447b639a2264c0747df
      sizeBytes: 495140160
    - names:
      - quay.io/openshift/okd-content@sha256:0159217f85c3402013e04be28cc4156ba48c12f92a70616ec89f3e9895b20254
      sizeBytes: 494654430
    - names:
      - quay.io/openshift/okd-content@sha256:ce71a63c3f4419b77e72bf8928a06a430b50cbf141bec8a7ff8dbd38bc81b69c
      sizeBytes: 492381581
    - names:
      - quay.io/openshift/okd-content@sha256:5d942a76a526c390e002d3e4391b533fb6bfe3f19ac0140137b4a4325556295c
      - quay.io/openshift/okd-content@sha256:efc47f5552964a88708262ede70bc9cd02a20cbeab000353fc8122c4baf0e287
      sizeBytes: 490586164
    - names:
      - quay.io/openshift/okd-content@sha256:8afdec954ef25fea78f69fb94a81254c377edddb968a5a730651e723189b7dfa
      - quay.io/openshift/okd-content@sha256:c637e6cc52bf132c2d23af5a993d213ce453306aaa7ff473840d5f66756d5cc7
      sizeBytes: 489474154
    - names:
      - quay.io/openshift/okd-content@sha256:2b2ce972fc3fffd384e94393e50e321168b8ea488fb62cf9059903acb0fa47d6
      sizeBytes: 483672938
    - names:
      - quay.io/openshift/okd-content@sha256:5a79d38a9613dc33b27cd2775d35613c0694c461b97135a04ebdbb7307532f47
      sizeBytes: 479923890
    - names:
      - quay.io/openshift/okd-content@sha256:fc90d9259e8f66cee8bd4e8ca187675331b2ef998e880cd04a0367e09ea8d035
      sizeBytes: 479827515
    - names:
      - quay.io/openshift/okd-content@sha256:3beac2d94e3006ace2cc937b691da6aa22ce0dba5c35d00bb497a7a5cd73068a
      - quay.io/openshift/okd-content@sha256:dd286a23ab14a2f262b67a76f23414a36bfbd7466f8ae94ba59dff659d3e270f
      sizeBytes: 478268055
    - names:
      - quay.io/openshift/okd-content@sha256:25d10ff56512914765c6d9f340d9ad6807cd10de9b804218de74a64f8be4b257
      sizeBytes: 472976545
    - names:
      - quay.io/openshift/okd-content@sha256:c8307b4291975183adc6a835e621e8c74d729ba611caa09c986f799104d24fc4
      sizeBytes: 461768877
    - names:
      - quay.io/openshift/okd-content@sha256:3392b9127b29b50e578618a36ce6ec3f3ae741337db5cfbb5618d36d1c235100
      - quay.io/openshift/okd-content@sha256:8e94d708c98dadc984806f21837b4a71c2b613e4965f0782c363c33387e70467
      sizeBytes: 461074249
    - names:
      - quay.io/openshift/okd-content@sha256:2cbc4bf38c29285c2b8e4052e90e77720b98c366046be96c1382867c3f4287b1
      sizeBytes: 455911503
    - names:
      - quay.io/openshift/okd-content@sha256:74499ce45829545bd83876e76a5266b1db64e256d9a68d7673f4fb93aef0a272
      - quay.io/openshift/okd-content@sha256:8cd4b7925c31290234220e019c33db0137ab0e45701d975ee8c2bd723b95f722
      sizeBytes: 446650003
    - names:
      - quay.io/openshift/okd-content@sha256:28856aae9e481fe763776df713b4b335632645687e6fd2e31619f76d11e0f47c
      sizeBytes: 446649853
    - names:
      - quay.io/openshift/okd-content@sha256:a7260990cd366b7cd382aa1acef71cf9f0d6afc0f22b7d36ddc9bb593ab76784
      sizeBytes: 441804242
    nodeInfo:
      architecture: amd64
      bootID: 408e4590-c3d0-4474-8dce-a30b4fb85340
      containerRuntimeVersion: cri-o://1.26.4
      kernelVersion: 6.5.5-200.fc38.x86_64
      kubeProxyVersion: v1.26.9+636f2be
      kubeletVersion: v1.26.9+636f2be
      machineID: 125e89b2a67b462eae379e78c26555eb
      operatingSystem: linux
      osImage: Fedora CoreOS 38.20231002.3.1
      systemUUID: 5f1f0d42-0903-e32e-31de-ba98297b2da3
kind: List
metadata:
  resourceVersion: ""
okdadmin commented 1 month ago

${CEPH_CSI_NAMESPACE} I installed rook-ceph-external! Ceph_csi_namespace don't have!

tongpu commented 1 month ago

${CEPH_CSI_NAMESPACE} I installed rook-ceph-external! Ceph_csi_namespace don't have!

Then please provide the output of oc get daemonset -n rook-ceph-external -o yaml.

Please enclose your output on code tags, which makes it easier to consume:

```yaml
this:
  is: an example
okdadmin commented 1 month ago

oc get daemonset -n rook-ceph-external -o yaml apiVersion: v1 items: [] kind: List metadata: resourceVersion: ""

okdadmin commented 1 month ago

daemonset.txt The rook-ceph-external only connection namespace. The operator namespace is rook-ceph.

tongpu commented 1 month ago

Thanks for the information. This proves what I expected. The daemonset serving the Ceph CSI driver is only scheduled to be started on the worker nodes and it not present on the control-plane nodes, because of that the openshift-etcd-backup pod can't be started on the master node, because it's unable to mount the volume you specified on the master nodes. To achieve this you would probably need to change the settings of your Ceph deployment to allow you to use the Ceph storage also on the control-plane nodes, or look for other storage alternatives (e.g. S3 or NFS).

okdadmin commented 1 month ago

Thank you for your answer! Do you have any idea what should be stated in rook-ceph?

tongpu commented 1 month ago

I don't have experience running rook-ceph, but probably configuring the tolerations, like described in deploy/examples/operator-openshift.yaml in the rook/rook repository should allow you to get the daemonset pods scheduled on the master nodes: https://github.com/rook/rook/blob/e84e782158dc137f3cab50a1e75d23b781594a8b/deploy/examples/operator-openshift.yaml#L270-L293

If your change is successful you should see 7 pods for you daemonsets in rook-ceph, instead of the 4 pods you currently have on your workers.