MON-3934: Clean up and parallelize some e2e tests for faster runs

[machine424]

MON-3934: Parallelize some readonly/non disruptive e2e tests for faster runs

Isolate specific tests to enable parallel execution

Enhance the resilience of some tests and fix those prone to errors.

Fix some tests that were running wrong checks.

Make some the tests idempotent to be easily debugged and run locally

• [ ] I added CHANGELOG entry for this change.
• [x] No user facing changes, so no entry in CHANGELOG was needed.

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: machine424

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/openshift/cluster-monitoring-operator/blob/master/OWNERS)~~ [machine424] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[openshift-ci-robot]

@machine424: This pull request references MON-3934 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-monitoring-operator/pull/2397): > > >* [ ] I added CHANGELOG entry for this change. >* [ ] No user facing changes, so no entry in CHANGELOG was needed. > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-monitoring-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

[machine424]

/retest

[openshift-ci-robot]

@machine424: This pull request references MON-3934 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-monitoring-operator/pull/2397): > > >* [ ] I added CHANGELOG entry for this change. >* [x] No user facing changes, so no entry in CHANGELOG was needed. > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-monitoring-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

[openshift-ci-robot]

@machine424: This pull request references MON-3934 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-monitoring-operator/pull/2397): > > > [MON-3934](https://issues.redhat.com//browse/MON-3934): Parallelize some readonly/non disruptive e2e tests for faster runs > > Isolate specific tests to enable parallel execution > > Enhance the resilience of some tests and fix those prone to errors. > > > > > > > >* [ ] I added CHANGELOG entry for this change. >* [x] No user facing changes, so no entry in CHANGELOG was needed. > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-monitoring-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

[openshift-ci-robot]

@machine424: This pull request references MON-3934 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.17.0" version, but no target version was set.

In response to [this](https://github.com/openshift/cluster-monitoring-operator/pull/2397): > [MON-3934](https://issues.redhat.com//browse/MON-3934): Parallelize some readonly/non disruptive e2e tests for faster runs > > Isolate specific tests to enable parallel execution > > Enhance the resilience of some tests and fix those prone to errors. > > Fix some tests that were running wrong checks. > > Make some the tests idempotent to be easily debugged and run locally > > > > > > > >* [ ] I added CHANGELOG entry for this change. >* [x] No user facing changes, so no entry in CHANGELOG was needed. > Instructions for interacting with me using PR comments are available [here](https://prow.ci.openshift.org/command-help?repo=openshift%2Fcluster-monitoring-operator). If you have questions or suggestions related to my behavior, please file an issue against the [openshift-eng/jira-lifecycle-plugin](https://github.com/openshift-eng/jira-lifecycle-plugin/issues/new) repository.

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

[openshift-ci[bot]]

@machine424: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/generate	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test generate
ci/prow/versions	00a6f602c9d30ed636028f231f7a21f7eb898116	link	false	/test versions
ci/prow/e2e-aws-ovn-techpreview	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test e2e-aws-ovn-techpreview
ci/prow/jsonnet-fmt	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test jsonnet-fmt
ci/prow/e2e-aws-ovn-single-node	00a6f602c9d30ed636028f231f7a21f7eb898116	link	false	/test e2e-aws-ovn-single-node
ci/prow/golangci-lint	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test golangci-lint
ci/prow/e2e-agnostic-operator	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test e2e-agnostic-operator
ci/prow/unit	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test unit
ci/prow/shellcheck	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test shellcheck
ci/prow/rules	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test rules
ci/prow/e2e-hypershift-conformance	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test e2e-hypershift-conformance
ci/prow/vendor	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test vendor
ci/prow/verify	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test verify
ci/prow/images	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test images
ci/prow/e2e-aws-ovn	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test e2e-aws-ovn
ci/prow/go-fmt	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test go-fmt
ci/prow/e2e-aws-ovn-upgrade	00a6f602c9d30ed636028f231f7a21f7eb898116	link	true	/test e2e-aws-ovn-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).