search

red-hat-storage / ocs-ci

https://ocs-ci.readthedocs.io/en/latest/
MIT License
109 stars 165 forks source link

test_run_io_multiple_pods is failing on ODF 4.16 on IBM Power #10025

Open Pooja-Soni78 opened 1 week ago

Pooja-Soni78 commented 1 week ago

Following test cases are failing on IBM P -

tests/functional/pv/pv_services/test_run_io_multiple_pods.py::TestIOMultiplePods::test_run_io_multiple_pods[CephBlockPool]
tests/functional/pv/pv_services/test_run_io_multiple_pods.py::TestIOMultiplePods::test_run_io_multiple_pods[CephFileSystem]

Error being -

        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,                                                                                 stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['oc', '--kubeconfig', '/root/auth/kubeconfig', '-n', 'namespace-test-31f34cf330064978bdd5d9a77', 'rsh', 'pod-test-rbd-a5a49c7faeac4cfd95fe903ce46', 'fio', '--name=fio-rand-readwrite', '--filename=/var/lib/www/html/fio-rand-readwrite', '--readwrite=randrw', '--bs=4K', '--direct=0', '--numjobs=1', '--time_based=1', '--runtime=60', '--size=4G', '--iodepth=4', '--invalidate=1', '--fsync_on_close=1', '--rwmixread=75', '--ioengine=libaio', '--rate=1m', '--rate_process=poisson', '--output-format=json']' timed out after 600 seconds

/usr/lib64/python3.9/subprocess.py:1178: TimeoutExpired
Pooja-Soni78 commented 1 week ago

Cluster details -

[root@pooj4160-new-2001-bastion-0 ~]# oc version
Client Version: 4.16.0-rc.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.16.0-rc.4
Kubernetes Version: v1.29.5+f6419fb
[root@pooj4160-new-2001-bastion-0 ~]#

sh-5.1# lscpu
Architecture:            ppc64le
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Model name:              POWER9 (architected), altivec supported
  Model:                 1.3 (pvr 004e 2103)
  Thread(s) per core:    8
  Core(s) per socket:    1
  Socket(s):             2
  Physical sockets:      16
  Physical chips:        1
  Physical cores/chip:   11
Virtualization features:
  Hypervisor vendor:     pHyp
  Virtualization type:   para
Caches (sum of all):
  L1d:                   128 KiB (4 instances)
  L1i:                   128 KiB (4 instances)
NUMA:
  NUMA node(s):          6
  NUMA node5 CPU(s):     0-7
  NUMA node6 CPU(s):
  NUMA node7 CPU(s):
  NUMA node9 CPU(s):
  NUMA node10 CPU(s):    8-15
  NUMA node11 CPU(s):
Vulnerabilities:
  Gather data sampling:  Not affected
  Itlb multihit:         Not affected
  L1tf:                  Mitigation; RFI Flush, L1D private per thread
  Mds:                   Not affected
  Meltdown:              Mitigation; RFI Flush, L1D private per thread
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec rstack overflow:  Not affected
  Spec store bypass:     Mitigation; Kernel entry/exit barrier (eieio)
  Spectre v1:            Mitigation; __user pointer sanitization, ori31 speculation barrier enabled
  Spectre v2:            Mitigation; Software count cache flush (hardware accelerated), Software link stack flush
  Srbds:                 Not affected
  Tsx async abort:       Not affected
sh-5.1#

sh-5.1# lsmem
RANGE                                  SIZE  STATE REMOVABLE BLOCK
0x0000000000000000-0x0000001fffffffff  128G online       yes 0-511

Memory block size:       256M
Total online memory:     128G
Total offline memory:      0B
sh-5.1#

test_run_io_multiple_pods[CephFileSystem]-1.zip test_run_io_multiple_pods[CephBlockPool]-1.zip