search

cloud-bulldozer / benchmark-operator

The Chuck Norris of cloud benchmarks
Apache License 2.0
282 stars 127 forks source link

CNV Perf CI[benchmark-runner]: Uperf workload failed in Pod/Kata/VM #777

Closed ebattat closed 2 years ago

ebattat commented 2 years ago

Benchmark-runner CNV CI failed in Uperf workload in last Nightly CI uperf-pod/vm does not ready

Can you please let me know what did you change that cause it ?

ebattat commented 2 years ago

Uperf is failing for pod and vm:

[   67.616827] cloud-init[749]: Installed:
[   67.617065] cloud-init[749]:   git-2.34.3-1.fc34.x86_64
[   67.617297] cloud-init[749]:   git-core-2.34.3-1.fc34.x86_64
[   67.617589] cloud-init[749]:   git-core-doc-2.34.3-1.fc34.noarch
[   67.617706] cloud-init[749]:   lksctp-tools-1.0.18-9.fc34.x86_64
[   67.617842] cloud-init[749]:   logrotate-3.18.0-3.fc34.x86_64
[   67.617969] cloud-init[749]:   perl-AutoLoader-5.74-477.fc34.noarch
[   67.618099] cloud-init[749]:   perl-B-1.80-477.fc34.x86_64
[   67.618232] cloud-init[749]:   perl-Carp-1.50-458.fc34.noarch
[   67.618368] cloud-init[749]:   perl-Class-Struct-0.66-477.fc34.noarch
[   67.618642] cloud-init[749]:   perl-Data-Dumper-2.174-460.fc34.x86_64
[   67.618795] cloud-init[749]:   perl-Digest-1.20-1.fc34.noarch
[   67.618933] cloud-init[749]:   perl-Digest-MD5-2.58-2.fc34.x86_64
[   67.619045] cloud-init[749]:   perl-DynaLoader-1.47-477.fc34.x86_64
[   67.619166] cloud-init[749]:   perl-Encode-4:3.15-462.fc34.x86_64
[   67.619285] cloud-init[749]:   perl-Errno-1.30-477.fc34.x86_64
[   67.619412] cloud-init[749]:   perl-Error-1:0.17029-5.fc34.noarch
[   67.619528] cloud-init[749]:   perl-Exporter-5.74-459.fc34.noarch
[   67.619652] cloud-init[749]:   perl-Fcntl-1.13-477.fc34.x86_64
[   67.619799] cloud-init[749]:   perl-File-Basename-2.85-477.fc34.noarch
[   67.619930] cloud-init[749]:   perl-File-Find-1.37-477.fc34.noarch
[   67.620057] cloud-init[749]:   perl-File-Path-2.18-2.fc34.noarch
[   67.620183] cloud-init[749]:   perl-File-Temp-1:0.231.100-2.fc34.noarch
[   67.620319] cloud-init[749]:   perl-File-stat-1.09-477.fc34.noarch
[   67.620581] cloud-init[749]:   perl-FileHandle-2.03-477.fc34.noarch
[   67.620727] cloud-init[749]:   perl-Getopt-Long-1:2.52-2.fc34.noarch
[   67.620841] cloud-init[749]:   perl-Getopt-Std-1.12-477.fc34.noarch
[   67.620982] cloud-init[749]:   perl-Git-2.34.3-1.fc34.noarch
[   67.621104] cloud-init[749]:   perl-HTTP-Tiny-0.078-1.fc34.noarch
[   67.621220] cloud-init[749]:   perl-IO-1.43-477.fc34.x86_64
[   67.621365] cloud-init[749]:   perl-IO-Socket-IP-0.41-3.fc34.noarch
[   67.621621] cloud-init[749]:   perl-IO-Socket-SSL-2.070-2.fc34.noarch
[   67.621765] cloud-init[749]:   perl-IPC-Open3-1.21-477.fc34.noarch
[   67.621902] cloud-init[749]:   perl-MIME-Base64-3.16-2.fc34.x86_64
[   67.622022] cloud-init[749]:   perl-Mozilla-CA-20211001-1.fc34.noarch
[   67.622140] cloud-init[749]:   perl-NDBM_File-1.15-477.fc34.x86_64
[   67.622259] cloud-init[749]:   perl-Net-SSLeay-1.90-2.fc34.x86_64
[   67.622388] cloud-init[749]:   perl-POSIX-1.94-477.fc34.x86_64
[   67.622503] cloud-init[749]:   perl-PathTools-3.78-459.fc34.x86_64
[   67.622620] cloud-init[749]:   perl-Pod-Escapes-1:1.07-458.fc34.noarch
[   67.622753] cloud-init[749]:   perl-Pod-Perldoc-3.28.01-459.fc34.noarch
[   67.622868] cloud-init[749]:   perl-Pod-Simple-1:3.42-2.fc34.noarch
[   67.622990] cloud-init[749]:   perl-Pod-Usage-4:2.01-2.fc34.noarch
[   67.623117] cloud-init[749]:   perl-Scalar-List-Utils-4:1.56-459.fc34.x86_64
[   67.623239] cloud-init[749]:   perl-SelectSaver-1.02-477.fc34.noarch
[   67.623356] cloud-init[749]:   perl-Socket-4:2.032-1.fc34.x86_64
[   67.623595] cloud-init[749]:   perl-Storable-1:3.21-458.fc34.x86_64
[   67.623731] cloud-init[749]:   perl-Symbol-1.08-477.fc34.noarch
[   67.623850] cloud-init[749]:   perl-Term-ANSIColor-5.01-459.fc34.noarch
[   67.623966] cloud-init[749]:   perl-Term-Cap-1.17-458.fc34.noarch
[   67.624097] cloud-init[749]:   perl-TermReadKey-2.38-9.fc34.x86_64
[   67.624220] cloud-init[749]:   perl-Text-ParseWords-3.30-458.fc34.noarch
[   67.624340] cloud-init[749]:   perl-Text-Tabs+Wrap-2021.0726-1.fc34.noarch
[   67.624474] cloud-init[749]:   perl-Time-Local-2:1.300-5.fc34.noarch
[   67.624501] cloud-init[749]:   perl-URI-5.09-1.fc34.noarch
[   67.624656] cloud-init[749]:   perl-base-2.27-477.fc34.noarch
[   67.625028] cloud-init[749]:   perl-constant-1.33-459.fc34.noarch
[   67.625219] cloud-init[749]:   perl-if-0.60.800-477.fc34.noarch
[   67.625368] cloud-init[749]:   perl-interpreter-4:5.32.1-477.fc34.x86_64
[   67.625548] cloud-init[749]:   perl-lib-0.65-477.fc34.x86_64
[   67.625913] cloud-init[749]:   perl-libnet-3.13-2.fc34.noarch
[   67.626225] cloud-init[749]:   perl-libs-4:5.32.1-477.fc34.x86_64
[   67.626363] cloud-init[749]:   perl-mro-1.23-477.fc34.x86_64
[   67.626789] cloud-init[749]:   perl-overload-1.31-477.fc34.noarch
[   67.626968] cloud-init[749]:   perl-overloading-0.02-477.fc34.noarch
[   67.627183] cloud-init[749]:   perl-parent-1:0.238-458.fc34.noarch
[   67.627322] cloud-init[749]:   perl-podlators-1:4.14-458.fc34.noarch
[   67.627606] cloud-init[749]:   perl-subs-1.03-477.fc34.noarch
[   67.627743] cloud-init[749]:   perl-vars-1.05-477.fc34.noarch
[   67.627868] cloud-init[749]:   redis-6.2.7-1.fc34.x86_64
[   67.627996] cloud-init[749]:   uperf-1.0.7-3.fc34.x86_64
[   67.628128] cloud-init[749]: Complete!
[   67.737673] cloud-init[749]: 0
[   67.741474] cloud-init[749]: ERR value is not an integer or out of range

Failing here

ebattat commented 2 years ago

Hi @rsevilla87, After investigation found the root cause: The new redis image cause the failure: Commit

I have tested on my local and it works perfect with old image bitnami/redis can we revert this fix back ?

    charts/benchmark-operator/values.yaml
    repository: quay.io/cloud-bulldozer/redis => bitnami/redis
rsevilla87 commented 2 years ago

Hi @rsevilla87, After investigation found the root cause: The new redis image cause the failure: Commit

I have tested on my local and it works perfect with old image bitnami/redis can we revert this fix back ?

    charts/benchmark-operator/values.yaml
    repository: quay.io/cloud-bulldozer/redis => bitnami/redis

Hi @rsevilla87, After investigation found the root cause: The new redis image cause the failure: Commit

I have tested on my local and it works perfect with old image bitnami/redis can we revert this fix back ?

    charts/benchmark-operator/values.yaml
    repository: quay.io/cloud-bulldozer/redis => bitnami/redis

As per our conversation in slack, can we close this issue?

ebattat commented 2 years ago

The current status is that uperf_vm is not working due redis db change. I think we should solve this issue before closing this issue.

ebattat commented 2 years ago

After short investigation with @mukrishn and @vishnuchalla, we found that: This issue is related to redis-cli version

Not able to reproduced in redis-cli version: $ redis-cli --version redis-cli 6.2.7

sh-4.4$ redis-cli -h 10.128.0.176  setnx 09165033 0
(integer) 0
sh-4.4$ redis-cli -h 10.128.0.176  incr 09165033 
(error) ERR value is not an integer or out of range

If this issue will be happened again need to install a specified version of redis here

sudo dnf install redis-6.2.7-1.fc34.x86_64

redis cli versions