search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
109 stars 166 forks source link

test_prometheus_rule_failures testcase is failing on ODF4.17 for IBM Power #10269

Open Arpanchak opened 2 months ago

Arpanchak commented 2 months ago

Following tier1 testcase is failing on IBM Power:

tests/functional/monitoring/prometheus/alerts/test_alerting_works.py::test_prometheus_rule_failures

error being:

07:38:53 - MainThread - ocs_ci.utility.utils - INFO  - testrun_name: OCS4-17-Downstream-OCP4-17-POWERVS-UPI-1AZ-RHCOS-LSO-3M-3W-tier1
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - ERROR  - upgrade mark does not exist
07:38:53 - MainThread - tests.conftest - INFO  - Checking for Ceph Health OK

logfile:

test_prometheus_rule_failures.log

ODF version and pods:

[root@arpan-zs-ecca-bastion-0 ~]# oc version
Client Version: 4.17.0-ec.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.17.0-ec.2
Kubernetes Version: v1.30.2
[root@arpan-zs-ecca-bastion-0 ~]# oc get csv -A
NAMESPACE                              NAME                                          DISPLAY                            VERSION               REPLACES   PHASE
openshift-local-storage                local-storage-operator.v4.17.0-202408051414   Local Storage                      4.17.0-202408051414
     Succeeded
openshift-operator-lifecycle-manager   packageserver                                 Package Server                     0.0.1-snapshot
     Succeeded
openshift-storage                      mcg-operator.v4.17.0-44.stable                NooBaa Operator                    4.17.0-44.stable
     Succeeded
openshift-storage                      ocs-client-operator.v4.17.0-44.stable         OpenShift Data Foundation Client   4.17.0-44.stable
     Succeeded
openshift-storage                      ocs-operator.v4.17.0-44.stable                OpenShift Container Storage        4.17.0-44.stable
     Succeeded
openshift-storage                      odf-csi-addons-operator.v4.17.0-44.stable     CSI Addons                         4.17.0-44.stable
     Succeeded
openshift-storage                      odf-operator.v4.17.0-44.stable                OpenShift Data Foundation          4.17.0-44.stable
     Succeeded
openshift-storage                      odf-prometheus-operator.v4.17.0-44.stable     Prometheus Operator                4.17.0-44.stable
     Succeeded
openshift-storage                      recipe.v4.17.0-44.stable                      Recipe                             4.17.0-44.stable
     Succeeded
openshift-storage                      rook-ceph-operator.v4.17.0-44.stable          Rook-Ceph                          4.17.0-44.stable
     Succeeded
[root@arpan-zs-ecca-bastion-0 ~]# oc get pods -n openshift-local-storage
NAME                                     READY   STATUS    RESTARTS   AGE
diskmaker-discovery-2j522                2/2     Running   0          72m
diskmaker-discovery-d4hfn                2/2     Running   0          72m
diskmaker-discovery-vnvv4                2/2     Running   0          72m
diskmaker-manager-6qjsm                  2/2     Running   0          76m
diskmaker-manager-st7nl                  2/2     Running   0          76m
diskmaker-manager-vwlgv                  2/2     Running   0          76m
local-storage-operator-75b96c7cc-snl6m   1/1     Running   0          88m
[root@arpan-zs-ecca-bastion-0 ~]# oc get localvolume -n openshift-local-storage
NAME         AGE
localblock   76m
[root@arpan-zs-ecca-bastion-0 ~]# oc get pods -n openshift-storage
NAME                                                              READY   STATUS      RESTARTS      AGE
csi-addons-csi-addons-controller-manager-55655f5794-lcwfv         2/2     Running     0             89m
csi-cephfsplugin-4xln2                                            3/3     Running     1 (71m ago)   72m
csi-cephfsplugin-7q2r7                                            3/3     Running     0             72m
csi-cephfsplugin-provisioner-665f87487-4dswq                      7/7     Running     0             72m
csi-cephfsplugin-provisioner-665f87487-nc65r                      7/7     Running     1 (71m ago)   72m
csi-cephfsplugin-zck95                                            3/3     Running     0             72m
csi-rbdplugin-hkhqg                                               4/4     Running     0             72m
csi-rbdplugin-provisioner-f5f4f44f4-cqdb8                         7/7     Running     0             72m
csi-rbdplugin-provisioner-f5f4f44f4-kr4c8                         7/7     Running     0             72m
csi-rbdplugin-qgs2r                                               4/4     Running     1 (71m ago)   72m
csi-rbdplugin-zcqfp                                               4/4     Running     0             72m
noobaa-core-0                                                     2/2     Running     0             70m
noobaa-db-pg-0                                                    1/1     Running     0             70m
noobaa-endpoint-8446dc44f-xzrpb                                   1/1     Running     0             68m
noobaa-operator-58764c7d45-9sn5z                                  1/1     Running     0             90m
ocs-client-operator-console-8dc65b87-htdqp                        1/1     Running     0             90m
ocs-client-operator-controller-manager-795749b6db-dk4x8           2/2     Running     0             90m
ocs-metrics-exporter-c79bd7b46-8w7k5                              3/3     Running     0             69m
ocs-operator-7dc6489dbb-tmdjs                                     1/1     Running     2 (90m ago)   90m
odf-console-ddd5cfc8c-n4vbt                                       1/1     Running     0             90m
odf-operator-controller-manager-6d66554b6f-xzw7c                  2/2     Running     0             90m
pod-test-cephfs-062c0c5b413a44d68517420f-1-deploy                 0/1     Completed   0             63s
pod-test-cephfs-062c0c5b413a44d68517420f-1-qkfdp                  1/1     Running     0             60s
rook-ceph-crashcollector-worker-0-f9b9dbf67-qd6h8                 1/1     Running     0             70m
rook-ceph-crashcollector-worker-1-77bb6bf998-46r45                1/1     Running     0             70m
rook-ceph-crashcollector-worker-2-7798c94bf7-zr9wf                1/1     Running     0             70m
rook-ceph-exporter-worker-0-6746f655fc-kbr8j                      1/1     Running     0             69m
rook-ceph-exporter-worker-1-7955b8684-8kdfh                       1/1     Running     0             70m
rook-ceph-exporter-worker-2-7945f994c9-9g6t6                      1/1     Running     0             70m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-855998cfxr7h2   2/2     Running     0             70m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-b6bd55b6t4rjv   2/2     Running     0             70m
rook-ceph-mgr-a-6f644757f7-5tvg4                                  3/3     Running     0             71m
rook-ceph-mgr-b-9df9c8d68-pt8fx                                   3/3     Running     0             71m
rook-ceph-mon-a-675498fb95-qpfjd                                  2/2     Running     0             72m
rook-ceph-mon-b-6f76948dcf-dsjqf                                  2/2     Running     0             71m
rook-ceph-mon-c-75dd4b6f5d-4trs7                                  2/2     Running     0             71m
rook-ceph-operator-64f5d7dd66-c6f8s                               1/1     Running     0             89m
rook-ceph-osd-0-7c459ff658-mscs8                                  2/2     Running     0             70m
rook-ceph-osd-1-ffbfc49c9-cdz5z                                   2/2     Running     0             70m
rook-ceph-osd-2-6775954fb6-k5hpp                                  2/2     Running     0             70m
rook-ceph-osd-prepare-204762bc197e18b8e46e6587184bb258-fthzq      0/1     Completed   0             71m
rook-ceph-osd-prepare-2784575d608c2e404089766940ec7f30-5cfh5      0/1     Completed   0             71m
rook-ceph-osd-prepare-a9bf5390d7af2e58b5ed8b04bf706523-hmh8p      0/1     Completed   0             71m
rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-767fdf9xhqpg   2/2     Running     0             70m
rook-ceph-tools-99687598f-bhvmr                                   1/1     Running     0             5m32s
ux-backend-server-748dc979ff-lxtfd                                2/2     Running     0             90m
[root@arpan-zs-ecca-bastion-0 ~]# oc get pvc -n openshift-storage
NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
   VOLUMEATTRIBUTESCLASS   AGE
db-noobaa-db-pg-0                          Bound    pvc-113a290d-3f3e-4688-9951-565c82772843   50Gi       RWO            ocs-storagecluster-ceph-rbd   <unset>                 70m
ocs-deviceset-localblock-0-data-08fqgc     Bound    local-pv-980ba154                          500Gi      RWO            localblock
   <unset>                 71m
ocs-deviceset-localblock-0-data-1t4pg4     Bound    local-pv-c91edb3a                          500Gi      RWO            localblock
   <unset>                 71m
ocs-deviceset-localblock-0-data-28g7fg     Bound    local-pv-de78d0c9                          500Gi      RWO            localblock
   <unset>                 71m
pvc-test-dcd77cd7c56547fc878a09604c16769   Bound    pvc-82294daf-dcc9-47bd-b77f-ce7f8e8d14a3   20Gi       RWO            ocs-storagecluster-cephfs     <unset>                 83s
[root@arpan-zs-ecca-bastion-0 ~]# oc get sc -n openshift-storage
NAME                                    PROVISIONER                             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
localblock                              kubernetes.io/no-provisioner            Delete          WaitForFirstConsumer   false                  77m
ocs-storagecluster-ceph-rbd (default)   openshift-storage.rbd.csi.ceph.com      Delete          Immediate              true                   71m
ocs-storagecluster-ceph-rgw             openshift-storage.ceph.rook.io/bucket   Delete          Immediate              false                  73m
ocs-storagecluster-cephfs               openshift-storage.cephfs.csi.ceph.com   Delete          Immediate              true                   70m
openshift-storage.noobaa.io             openshift-storage.noobaa.io/obc         Delete          Immediate              false                  69m
[root@arpan-zs-ecca-bastion-0 ~]# oc get storagecluster -n openshift-storage
NAME                 AGE   PHASE   EXTERNAL   CREATED AT             VERSION
ocs-storagecluster   73m   Ready              2024-08-06T15:35:14Z   4.17.0
[root@arpan-zs-ecca-bastion-0 ~]# oc get cephcluster -n openshift-storage
NAME                             DATADIRHOSTPATH   MONCOUNT   AGE   PHASE   MESSAGE                        HEALTH      EXTERNAL   FSID
ocs-storagecluster-cephcluster   /var/lib/rook     3          73m   Ready   Cluster created successfully   HEALTH_OK              4e892795-9901-4198-bf0c-005dc425e2e2
[root@arpan-zs-ecca-bastion-0 ~]#
shyRozen commented 3 days ago

Hi @DanielOsypenko Please take a look. It is also failing all 3-4 tier 1 tests. ocs-ci results for OCS4-17-Downstream-OCP4-17-VSPHERE6-IPI-1AZ-RHCOS-VSAN-3M-3W-tier1-or-tier_after_upgrade-post-upgrade ocs-ci results for OCS4-17-Downstream-OCP4-17-AWS-IPI-3AZ-RHCOS-3M-3W-tier1-or-tier_after_upgrade-post-upgrade ocs-ci results for OCS4-17-Downstream-OCP4-17-VSPHERE6-UPI-Disconnected-1AZ-RHCOS-VSAN-3M-3W-tier1