[WIP] DNM: check ip forwarding during UDN tests

trozet commented 3 weeks ago

We suspect some other component in openshift is turning on global ip forwarding during CI runs.

trozet commented 3 weeks ago

/test e2e-gcp-ovn-techpreview

openshift-ci[bot] commented 3 weeks ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: trozet

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[test/extended/networking/OWNERS](https://github.com/openshift/origin/blob/master/test/extended/networking/OWNERS)~~ [trozet] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

trozet commented 3 weeks ago

/test e2e-gcp-ovn-techpreview

trozet commented 3 weeks ago

ip forwarding is being enabled by something... https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/29239/pull-ci-openshift-origin-master-e2e-gcp-ovn-techpreview/1851056906940977152

now we got to figure out who is doing it

trozet commented 3 weeks ago

this could be a GCP thing: https://github.com/kubernetes-sigs/cluster-api-provider-gcp/blob/main/config/crd/bases/infrastructure.cluster.x-k8s.io_gcpmachinetemplates.yaml#L231

lets try aws

trozet commented 3 weeks ago

/test e2e-aws-ovn-techpreview

openshift-ci[bot] commented 3 weeks ago

@trozet: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

/test e2e-aws-jenkins
/test e2e-aws-ovn-edge-zones
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-image-registry
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial
/test e2e-gcp-ovn
/test e2e-gcp-ovn-builds
/test e2e-gcp-ovn-image-ecosystem
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test images
/test lint
/test unit
/test verify
/test verify-deps

The following commands are available to trigger optional jobs:

/test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade-rollback
/test e2e-agnostic-ovn-cmd
/test e2e-aws
/test e2e-aws-csi
/test e2e-aws-disruptive
/test e2e-aws-etcd-certrotation
/test e2e-aws-etcd-recovery
/test e2e-aws-ovn
/test e2e-aws-ovn-cgroupsv2
/test e2e-aws-ovn-etcd-scaling
/test e2e-aws-ovn-kube-apiserver-rollout
/test e2e-aws-ovn-kubevirt
/test e2e-aws-ovn-single-node
/test e2e-aws-ovn-single-node-serial
/test e2e-aws-ovn-single-node-techpreview
/test e2e-aws-ovn-single-node-techpreview-serial
/test e2e-aws-ovn-single-node-upgrade
/test e2e-aws-ovn-upgrade
/test e2e-aws-ovn-upgrade-rollback
/test e2e-aws-ovn-upi
/test e2e-aws-ovn-virt-techpreview
/test e2e-aws-proxy
/test e2e-azure
/test e2e-azure-ovn-etcd-scaling
/test e2e-azure-ovn-upgrade
/test e2e-baremetalds-kubevirt
/test e2e-external-aws
/test e2e-external-aws-ccm
/test e2e-external-vsphere-ccm
/test e2e-gcp-csi
/test e2e-gcp-disruptive
/test e2e-gcp-fips-serial
/test e2e-gcp-ovn-etcd-scaling
/test e2e-gcp-ovn-rt-upgrade
/test e2e-gcp-ovn-techpreview
/test e2e-gcp-ovn-techpreview-serial
/test e2e-metal-ipi-ovn
/test e2e-metal-ipi-ovn-dualstack
/test e2e-metal-ipi-ovn-dualstack-local-gateway
/test e2e-metal-ipi-ovn-kube-apiserver-rollout
/test e2e-metal-ipi-serial
/test e2e-metal-ipi-serial-ovn-ipv6
/test e2e-metal-ipi-virtualmedia
/test e2e-metal-ovn-single-node-live-iso
/test e2e-metal-ovn-single-node-with-worker-live-iso
/test e2e-openstack-ovn
/test e2e-openstack-serial
/test e2e-vsphere
/test e2e-vsphere-ovn-dualstack-primaryv6
/test e2e-vsphere-ovn-etcd-scaling
/test okd-e2e-gcp
/test okd-scos-images

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-origin-master-e2e-agnostic-ovn-cmd
pull-ci-openshift-origin-master-e2e-aws-csi
pull-ci-openshift-origin-master-e2e-aws-ovn-cgroupsv2
pull-ci-openshift-origin-master-e2e-aws-ovn-edge-zones
pull-ci-openshift-origin-master-e2e-aws-ovn-fips
pull-ci-openshift-origin-master-e2e-aws-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-master-e2e-aws-ovn-microshift
pull-ci-openshift-origin-master-e2e-aws-ovn-microshift-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-serial
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node-upgrade
pull-ci-openshift-origin-master-e2e-aws-ovn-upgrade
pull-ci-openshift-origin-master-e2e-gcp-csi
pull-ci-openshift-origin-master-e2e-gcp-ovn
pull-ci-openshift-origin-master-e2e-gcp-ovn-rt-upgrade
pull-ci-openshift-origin-master-e2e-gcp-ovn-upgrade
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-ipv6
pull-ci-openshift-origin-master-e2e-metal-ipi-ovn-kube-apiserver-rollout
pull-ci-openshift-origin-master-e2e-openstack-ovn
pull-ci-openshift-origin-master-images
pull-ci-openshift-origin-master-lint
pull-ci-openshift-origin-master-unit
pull-ci-openshift-origin-master-verify
pull-ci-openshift-origin-master-verify-deps

In response to [this](https://github.com/openshift/origin/pull/29239#issuecomment-2444234768): >/test e2e-aws-ovn-techpreview Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.

trozet commented 3 weeks ago

/test e2e-aws-ovn-single-node-techpreview

trozet commented 3 weeks ago

/test e2e-aws-ovn-single-node-techpreview-serial

trozet commented 3 weeks ago

we think the root cause is bridge CNI:

https://github.com/containernetworking/plugins/blob/main/pkg/ip/ipforward_linux.go#L24-L26

Will confirm and the propose a change upstream.

openshift-ci[bot] commented 3 weeks ago

@trozet: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws-ovn-single-node-techpreview	fa52becdc6896c4129e4cfc5151fc89c85b32375	link	false	`/test e2e-aws-ovn-single-node-techpreview`
ci/prow/e2e-aws-ovn-single-node	fa52becdc6896c4129e4cfc5151fc89c85b32375	link	false	`/test e2e-aws-ovn-single-node`
ci/prow/e2e-aws-ovn-single-node-serial	fa52becdc6896c4129e4cfc5151fc89c85b32375	link	false	`/test e2e-aws-ovn-single-node-serial`
ci/prow/e2e-gcp-ovn-techpreview	fa52becdc6896c4129e4cfc5151fc89c85b32375	link	false	`/test e2e-gcp-ovn-techpreview`
ci/prow/e2e-aws-ovn-serial	fa52becdc6896c4129e4cfc5151fc89c85b32375	link	true	`/test e2e-aws-ovn-serial`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).

openshift-trt-bot commented 3 weeks ago

Job Failure Risk Analysis for sha: fa52becdc6896c4129e4cfc5151fc89c85b32375

Job Name	Failure Risk
pull-ci-openshift-origin-master-e2e-aws-ovn-single-node	Medium [sig-node] static pods should start after being created This test has passed 87.72% of 57 runs on jobs ['periodic-ci-openshift-release-master-nightly-4.18-e2e-aws-ovn-single-node'] in the last 14 days. Open Bugs Static pod controller pods sometimes fail to start [kube-controller-manager] Static pod controller pods sometimes fail to start [etcd]

openshift / origin

[WIP] DNM: check ip forwarding during UDN tests #29239