openshift / svt

Apache License 2.0
123 stars 105 forks source link

debug cronjob failure on Prow #780

Closed qiliRedHat closed 8 months ago

qiliRedHat commented 8 months ago

Run relialibity-v2 on prow, I saw 2 differences compared to running on jumphost

  1. cronjob failure

    2024-01-11 09:36:38,694 - INFO - [User: testuser-16] [Task: /tmp/svt/reliability-v2/tasks/script/cronjob.sh -n 10]: will be run
    2024-01-11 09:36:38,694 - INFO - => /tmp/svt/reliability-v2/tasks/script/cronjob.sh -n 10
    2024-01-11 09:36:38,699 - ERROR - /tmp/svt/reliability-v2/tasks/script/cronjob.sh -n 10 
    2 : /tmp/svt/reliability-v2/tasks/script/cronjob.sh: 20: function: not found
    Usage: cronjob.sh [-n <number of cronjobs>] [-s <schedule>] -d
    -n <number>                  : Number of cronjobs to be created. Default is 1.
    -s <schedule>                : Schedule for the cronjob. Default is '*/1 * * * *'
    -d                           : Delete all cronjobs and
    /tmp/svt/reliability-v2/tasks/script/cronjob.sh: 30: Syntax error: "}" unexpected

    When I manually run this script on the prow pod, it did not have this issue. I tired to run the same python version as the prow pod 3.9.9 on my local, I can not reproduce the issue too. Added 'set -x' to get more info to debug

  2. load app failure rate is higher on prow pod (7% in aws 8 hours run), it could be related to the prow pod's network config or condition. Reduce the load test client number from 10 to 1 to see if the failure rate can get down.

openshift-ci[bot] commented 8 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: qiliRedHat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/svt/blob/master/OWNERS)~~ [qiliRedHat] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
openshift-ci[bot] commented 8 months ago

@qiliRedHat: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
liqcui commented 8 months ago

/lgtm