openshift / origin

Conformance test suite for OpenShift
http://www.openshift.org
Apache License 2.0
8.44k stars 4.69k forks source link

OCPBUGS-26498: Add test for UpgradeValidation contention #28710

Open gcs278 opened 2 weeks ago

gcs278 commented 2 weeks ago

Add "The HAProxy router converges when multiple routers are writing conflicting upgrade validation status" test which validates router converge when writing conflicting status in a scenario that uses multiple conditions.

Testing contention at a condition-level is important because it exercises a slightly different logic path.

openshift-ci-robot commented 2 weeks ago

@gcs278: This pull request references Jira Issue OCPBUGS-26498, which is valid.

3 validation(s) were run on this bug * bug is open, matching expected state (open) * bug target version (4.16.0) matches configured target version for branch (4.16.0) * bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact: /cc @ShudiLi

The bug has been updated to refer to the pull request using the external bug tracker.

In response to [this](https://github.com/openshift/origin/pull/28710): >Add "The HAProxy router converges when multiple routers are writing conflicting upgrade validation status" test which validates router converge when writing conflicting status in a scenario that uses multiple conditions. > >Testing contention at a condition-level is important because it exercises a slightly different logic path. > >WIP because I need https://github.com/openshift/router/pull/575 merged. Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Forigin). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.
openshift-ci[bot] commented 2 weeks ago

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: gcs278 Once this PR has been reviewed and has the lgtm label, please assign miciah for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[test/extended/router/OWNERS](https://github.com/openshift/origin/blob/master/test/extended/router/OWNERS)** - **[test/extended/util/annotate/generated/OWNERS](https://github.com/openshift/origin/blob/master/test/extended/util/annotate/generated/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
openshift-trt-bot commented 2 weeks ago

Job Failure Risk Analysis for sha: d40ce53206a8dc32b5421c205d93b0b606c905d3

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-fips IncompleteTests
Tests for this run (98) are below the historical average (1540): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node Low
[sig-arch] events should not repeat pathologically for ns/openshift-etcd
This test has passed 77.55% of 49 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.16-e2e-aws-ovn-single-node'] in the last 14 days.
openshift-trt-bot commented 1 week ago

Job Failure Risk Analysis for sha: 4fd7700474e0c64cf6fb038b8f7938c517b00070

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade High
[sig-apps] job-upgrade
This test has passed 100.00% of 27 runs on jobs ['periodic-ci-openshift-release-master-ci-4.16-e2e-aws-ovn-upgrade'] in the last 14 days.
frobware commented 1 week ago

https://github.com/openshift/router/pull/575 has merged.

/test all

gcs278 commented 1 week ago

All other test failures weren't related to our new test The HAProxy router converges when multiple routers are writing conflicting upgrade validation status. We should just ignore them since they are not required and we don't impact them with this PR.

The test is in e2e-gcp-ovn and it was succesfull. Spinning a couple more times. /test e2e-gcp-ovn

gcs278 commented 1 week ago

Build image tests from the repository - not related. /test e2e-gcp-ovn

gcs278 commented 1 week ago

The HAProxy router converges when multiple routers are writing conflicting upgrade validation status. test also gets run in the e2e-aws-ovn-fips test: /test e2e-aws-ovn-fips

gcs278 commented 1 week ago

Image builds are failing. Appears like infrastructure issue.

openshift-trt-bot commented 1 week ago

Job Failure Risk Analysis for sha: f4d848cc06e6ac41a0662139123e1371598726ad

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade High
[sig-apps] job-upgrade
This test has passed 100.00% of 28 runs on jobs ['periodic-ci-openshift-release-master-ci-4.16-e2e-aws-ovn-upgrade'] in the last 14 days.
pull-ci-openshift-origin-master-e2e-gcp-ovn IncompleteTests
Tests for this run (16) are below the historical average (1718): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-gcp-csi IncompleteTests
Tests for this run (25) are below the historical average (692): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
gcs278 commented 1 week ago

@frobware found a flake. holding until we figure it out. /hold

gcs278 commented 1 week ago

/assign @frobware

openshift-trt-bot commented 1 week ago

Job Failure Risk Analysis for sha: 3aeab200aa54c7202a8cc0a4e5f50826e9c774f1

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-csi IncompleteTests
Tests for this run (25) are below the historical average (542): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
openshift-trt-bot commented 1 day ago

Job Failure Risk Analysis for sha: ae38bf21ee6d8814ac2d8aac16c5d3b7fe2f6cd3

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade IncompleteTests
Tests for this run (20) are below the historical average (2151): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial IncompleteTests
Tests for this run (19) are below the historical average (714): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node IncompleteTests
Tests for this run (19) are below the historical average (1598): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
gcs278 commented 16 hours ago

CI Infrastructure failures /retest

openshift-trt-bot commented 13 hours ago

Job Failure Risk Analysis for sha: 966f2eaea5ad28d9a9a8f611f4d16142ce923023

Job Name Failure Risk
pull-ci-openshift-origin-master-e2e-gcp-ovn-builds IncompleteTests
Tests for this run (25) are below the historical average (781): IncompleteTests (not enough tests ran to make a reasonable risk analysis; this could be due to infra, installation, or upgrade problems)
openshift-ci[bot] commented 9 hours ago

@gcs278: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn-upgrade f4d32eacbcd703dbf07a39d39dfbf50e6c5a7e26 link true /test e2e-gcp-ovn-upgrade
ci/prow/e2e-metal-ipi-sdn f4d32eacbcd703dbf07a39d39dfbf50e6c5a7e26 link false /test e2e-metal-ipi-sdn
ci/prow/e2e-gcp-ovn-builds f4d32eacbcd703dbf07a39d39dfbf50e6c5a7e26 link true /test e2e-gcp-ovn-builds
ci/prow/e2e-aws-ovn-single-node-upgrade f4d32eacbcd703dbf07a39d39dfbf50e6c5a7e26 link false /test e2e-aws-ovn-single-node-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).