search

kubernetes-sigs / cluster-api

Home for Cluster API, a subproject of sig-cluster-lifecycle
https://cluster-api.sigs.k8s.io
Apache License 2.0
3.58k stars 1.31k forks source link

clusterctl upgrade tests are flaky #9688

Closed killianmuldoon closed 2 months ago

killianmuldoon commented 1 year ago

The clusterctl upgrade tests have been significantly flaky in the last couple of weeks, with flakes occurring on main release-1.4 and release-1.5.

The flakes are occurring across many forms of the clusterctl upgrade tests including v0.4=>current, v1.3=>current and v1.0=>current.

The failures take a number of forms, including but not limited to:

There's an overall triage for tests with clusterctl upgrades in the name here: https://storage.googleapis.com/k8s-triage/index.html?date=2023-11-08&job=.*-cluster-api-.*&test=.*clusterctl%20upgrades.*&xjob=.*-provider-.*

/kind flake

killianmuldoon commented 1 year ago

@kubernetes-sigs/cluster-api-release-team These flakes are very disruptive to the test signal right now. It would be great if someone could prioritize investigating and fixing them out ahead of the releases.

/triage accepted

killianmuldoon commented 1 year ago

/help

k8s-ci-robot commented 1 year ago

@killianmuldoon: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/9688): >/help Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
killianmuldoon commented 1 year ago

Note that each branch has a different number of variants, enumerated below, of this test which may be responsible for some unevenness in the signal:

adilGhaffarDev commented 12 months ago

I am looking into this one.

furkatgofurov7 commented 12 months ago

I will be pairing up with @adilGhaffarDev on this one since it is happening more frequently.

/assign @adilGhaffarDev

adilGhaffarDev commented 11 months ago

Adding a bit more explanation regarding failures. We have three failures in clusterctl upgrade:

sbueringer commented 11 months ago

I have looked into logs and I am seeing this error:

{"ts":1700405055471.4797,"caller":"builder/webhook.go:184","msg":"controller-runtime/builder: Conversion webhook enabled","v":0,"GVK":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=DockerClusterTemplate"} {"ts":1700405055471.7637,"caller":"builder/webhook.go:139","msg":"controller-runtime/builder: skip registering a mutating webhook, object does not implement admission.Defaulter or WithDefaulter wasn't called","v":0,"GVK":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=DockerMachinePool"} {"ts":1700405055472.0557,"caller":"builder/webhook.go:168","msg":"controller-runtime/builder: skip registering a validating webhook, object does not implement admission.Validator or WithValidator wasn't called","v":0,"GVK":"infrastructure.cluster.x-k8s.io/v1beta1, Kind=DockerMachinePool"}

This is not an error. These are just info messages that surface that we are calling ctrl.NewWebhookManagedBy(mgr).For(c).Complete() for an object that has no validating or defaulting webhooks (we still get the same on main as we should)

adilGhaffarDev commented 9 months ago

Update on this issue. I am not seeing following flakes anymore:

failed to discovery ownerGraph types flake is still happening but only when upgrading from (v0.4=>current)

Ref: https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#3c64d10ff3eda504da75

sbueringer commented 9 months ago

@adilGhaffarDev So the clusterctl upgrade test is 100% stable apart from "failed to discovery ownerGraph types flake is still happening but only when upgrading from (v0.4=>current)"?

Ref: https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#3c64d10ff3eda504da75

Is not showing anything for me

adilGhaffarDev commented 9 months ago

@adilGhaffarDev So the clusterctl upgrade test is 100% stable apart from "failed to discovery ownerGraph types flake is still happening but only when upgrading from (v0.4=>current)"?

sorry for the bad link, here is more persitent link: https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&test=clusterctl%20upgrades%20&xjob=.*-provider-.*

Maybe not 100% stable there are very minor flakes that happen sometimes. But failed to find releases and exec.ExitError are not happening anymore.

sbueringer commented 9 months ago

@adilGhaffarDev exec.ExitError does not occur anymore because I improved the error output here: https://github.com/kubernetes-sigs/cluster-api/blob/adce02023c22d8681eb4ff5e0ae8df9eee5b8420/test/framework/cluster_proxy.go#L258

(https://github.com/kubernetes-sigs/cluster-api/pull/9737/files)

That doesn't mean the underlying errors are fixed unfortunately.

adilGhaffarDev commented 9 months ago

@adilGhaffarDev exec.ExitError does not occur anymore because I improved the error output here:

exec.ExitError was happening at step: INFO: Applying the cluster template yaml to the cluster I don't see any failure that is happening at the same step where exec.ExitError was happening. Do you see any failure on triage that is related to that? I am unable to find it.

sbueringer commented 9 months ago

Sounds good! Nope I didn't see any. Just wanted to clarify that the errors would look different now. But if the same step works now, it should be fine.

Just not sure what changed as I don't remember fixing/changing anything there.

adilGhaffarDev commented 9 months ago

Just not sure what changed as I don't remember fixing/changing anything there.

This is the new error that was happening after your PR, it seems like it stopped happening after 07-12-2023. https://storage.googleapis.com/k8s-triage/index.html?date=2023-12-10&job=.*-cluster-api-.*&xjob=.*-provider-.*#6710a9c85a9bbdb4d278

Only PR on 07-12-2023 that might have fixed this seemed to be this one: https://github.com/kubernetes-sigs/cluster-api/pull/9819 , but I am not sure.

sbueringer commented 9 months ago

9819 Should not be related. This func is called later in clusterctl_upgrade.go (l.516). While the issue happens in l.389.

So this is the error we get there

{Expected success, but got an error:
    <*errors.fundamental | 0xc000912948>: 
    exit status 1: stderr: 
    {
        msg: "exit status 1: stderr: ",
        stack: [0x1f3507a, 0x2010aa2, 0x84e4db, 0x862a98, 0x4725a1],
    } failed [FAILED] Expected success, but got an error:
    <*errors.fundamental | 0xc000912948>: 
    exit status 1: stderr: 
    {
        msg: "exit status 1: stderr: ",
        stack: [0x1f3507a, 0x2010aa2, 0x84e4db, 0x862a98, 0x4725a1],
    }

This is the corresponding output (under "open stdout")

Running kubectl apply --kubeconfig /tmp/e2e-kubeconfig3133952171 -f -
stderr:
Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

stdout:

So looks like the mgmt cluster was not reachable.

Thx for digging into this. I would say let's ignore this error for now as it's not occurring anymore. Good enough for me to know the issue stopped happening (I assumed it might be still there and just looks different).

adilGhaffarDev commented 9 months ago

Little more explanation to clusterctl upgrade failure. Now we are seeing only one flake when upgrading from 0.4->1.4 or 0.4->1.5, as mentioned before. Its failing with following error:

failed to discovery ownerGraph types: action failed after 9 attempts: failed to list "infrastructure.cluster.x-k8s.io/v1beta1, Kind=DockerCluster" resources: conversion webhook for infrastructure.cluster.x-k8s.io/v1alpha4, Kind=DockerCluster failed: Post "https://capd-webhook-service.capd-system.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority

This failure happens in post upgrade step, where we are are calling ValidateOwnerReferencesOnUpdate . We have this post upgrade step only when we upgrading from v1alpha to v1beta. I believe @killianmuldoon you have worked on it, can you check this when you get time.

chrischdi commented 8 months ago

Little more explanation to clusterctl upgrade failure. Now we are seeing only one flake when upgrading from 0.4->1.4 or 0.4->1.5, as mentioned before. Its failing with following error:

failed to discovery ownerGraph types: action failed after 9 attempts: failed to list "infrastructure.cluster.x-k8s.io/v1beta1, Kind=DockerCluster" resources: conversion webhook for infrastructure.cluster.x-k8s.io/v1alpha4, Kind=DockerCluster failed: Post "https://capd-webhook-service.capd-system.svc:443/convert?timeout=30s": x509: certificate signed by unknown authority

This failure happens in post upgrade step, where we are are calling ValidateOwnerReferencesOnUpdate . We have this post upgrade step only when we upgrading from v1alpha to v1beta. I believe @killianmuldoon you have worked on it, can you check this when you get time.

🤔 : may be helpful to collect cert-manager resources + logs to analyse this. Or is this locally reproducible?

adilGhaffarDev commented 8 months ago

🤔 : may be helpful to collect cert-manager resources + logs to analyse this. Or is this locally reproducible?

I haven't been able to reproduce locally. I have ran it multiple times.

chrischdi commented 8 months ago

Some observation via #10193 :

I0223 19:20:31.564028       1 controller.go:162] "re-queuing item due to optimistic locking on resource" logger="cert-manager.certificates-key-manager" key="capd-system/capd-serving-cert" error="Operation cannot be fulfilled on certificates.cert-manager.io \"capd-serving-cert\": the object has been modified; please apply your changes to the latest version and try again"

Source: https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/10193/pull-cluster-api-e2e-blocking-release-1-5/1761093535257333760/artifacts/5/clusters/clusterctl-upgrade-k13xsy/logs/cert-manager/cert-manager/cert-manager-6bbb455dc9-p88kt/cert-manager-controller.log

Maybe related cert-manager issue: https://github.com/cert-manager/cert-manager/issues/6464

Edit updated #10193 to now hopefully collect the cert-manager CRs. Maybe we can implement something which waits for the certificates to be ready or similar.

chrischdi commented 8 months ago

Maybe: worth taking a look if there could be an improvement to the clusterctl upgrade flow. The question is:

chrischdi commented 8 months ago

Testing a potential fix: https://github.com/kubernetes-sigs/cluster-api/pull/10193/commits/6f73c3a1d27c42c6526e653c1d281d5a1456043a

at https://github.com/kubernetes-sigs/cluster-api/pull/10193

chrischdi commented 8 months ago

Fix is here for the failed to discovery ownerGraph types error:

this should catch all x509: certificate signed by unknown authority / failed to discovery ownerGraph types errors which occur in clusterctl_upgrade tests related to conversion webhooks.

Should get cherry-picked to all supported branches.

link for ownergraph errors

link for all x509 errors to check occurencies of the flake and confirm it is fixed.

adilGhaffarDev commented 8 months ago

@chrischdi thank you for working on it, now we are not seeing this flake too much, nice work. On k8s triage I can see that now ownergraph flake is only happening in (v0.4=>v1.6=>current) tests, the other flakes seem to be fixed or they are much less flaky. ref: https://storage.googleapis.com/k8s-triage/index.html?job=.*-cluster-api-.*&xjob=.*-provider-.*#4f4c67c927112191922f

chrischdi commented 8 months ago

@chrischdi thank you for working on it, now we are not seeing this flake too much, nice work. On k8s triage I can see that now ownergraph flake is only happening in (v0.4=>v1.6=>current) tests, the other flakes seem to be fixed or they are much less flaky. ref: storage.googleapis.com/k8s-triage/index.html?job=.-cluster-api-.&xjob=.-provider-.#4f4c67c927112191922f

Note: this is a different flake, not directly ownergraph but similar. It happens at a different place though.

https://github.com/kubernetes-sigs/cluster-api/blob/487ed95f7842159f55727d510742ddcef9a900c9/test/e2e/clusterctl_upgrade.go#L553-L568

We could propably also ignore the x509 errors here and ensure that the last try in Consistently succeeded (by storing and checking the last error outside of Consistently)

sbueringer commented 8 months ago

We could propably also ignore the x509 errors here and ensure that the last try in Consistently succeeded (by storing and checking the last error outside of Consistently)

We could also add an Eventually before to wait until the List call works and then keep the Consistently the same

Btw, thx folks, really nice work on this issue!

adilGhaffarDev commented 8 months ago

We could also add an Eventually before to wait until the List call works and then keep the Consistently the same

I will open a PR with your suggestion

adilGhaffarDev commented 7 months ago

10301 did not fix the issue, failure is still there for (v0.4=>v1.6=>current) tests. I will try to reproduce it locally.

fabriziopandini commented 7 months ago

/priority important-soon

chrischdi commented 6 months ago

I implemented a fix at #10469 which should fix the situation.

k8s-triage-robot commented 3 months ago

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

fabriziopandini commented 3 months ago

/triage accepted Let's also consider to close and open a new one with the current state

cahillsf commented 2 months ago

/triage accepted Let's also consider to close and open a new one with the current state

agreed, i think a new issue would be helpful. the incoming release CI team can prioritize this. @chandankumar4 @adilGhaffarDev @Sunnatillo is there a summary of where we stand? if not, ill take a shot at refreshing the investigation and can open the new issue

seems like we do have flakes on main with a few different patterns shown for today: https://storage.googleapis.com/k8s-triage/index.html?date=2024-08-17&job=.*periodic-cluster-api-e2e.*&test=.*clusterctl%20upgrades.*

Sunnatillo commented 2 months ago

/triage accepted Let's also consider to close and open a new one with the current state

agreed, i think a new issue would be helpful. the incoming release CI team can prioritize this. @chandankumar4 @adilGhaffarDev @Sunnatillo is there a summary of where we stand? if not, ill take a shot at refreshing the investigation and can open the new issue

seems like we do have flakes on main with a few different patterns shown for today: https://storage.googleapis.com/k8s-triage/index.html?date=2024-08-17&job=.*periodic-cluster-api-e2e.*&test=.*clusterctl%20upgrades.*

From my observation I would say there are two main flakes that are occuring in clusterctl upgrade tests:

  1. https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-mink8s-main/1824761651627298816

    {Expected success, but got an error:
<errors.aggregate | len:3, cap:4>: 
[Internal error occurred: failed calling webhook "default.dockercluster.infrastructure.cluster.x-k8s.io": failed to call webhook: Post "https://capd-webhook-service.capd-system.svc:443/mutate-infrastructure-cluster-x-k8s-io-v1beta1-dockercluster?timeout=10s": dial tcp 10.96.44.105:443: connect: connection refused, Internal error occurred: failed calling webhook "validation.dockermachinetemplate.infrastructure.cluster.x-k8s.io": failed to call webhook: Post "https://capd-webhook-service.capd-system.svc:443/validate-infrastructure-cluster-x-k8s-io-v1beta1-dockermachinetemplate?timeout=10s": dial tcp 10.96.44.105:443: connect: connection refused]

  2. https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/periodic-cluster-api-e2e-mink8s-release-1-8/1823102356049367040 

    Timed out after 300.001s.
Timed out waiting for all Machines to exist
Expected
<int64>: 0
to equal
<int64>: 2
[FAILED] Timed out after 300.001s.
Timed out waiting for all Machines to exist
Expected
<int64>: 0
to equal
<int64>: 2
In [It] at: /home/prow/go/src/sigs.k8s.io/cluster-api/test/e2e/clusterctl_upgrade.go:500

First flake happening more often and when upgrading from latest versions, second flake happening mostly when uplifting from older releases.

I agree that we should close this issue and open new one separately for each flake.

sbueringer commented 2 months ago

@chrischdi Was looking into some of these issues and is about to write an update here. Let's wait for that before closing this issue

chrischdi commented 2 months ago

Sorry folks, took longer than expected.

According to aggregated failures of the last two weeks, we still have some flakyness on our clusterctl upgrade tests.

But it looks like none of them are the ones in the initial post:

Link to check if messages changed or we have new flakes on clusterctl upgrade tests: here

cahillsf commented 2 months ago

thank you for putting this together @chrischdi -- you mind if i copy paste this refreshed summary into a new issue and close the current one?

sbueringer commented 2 months ago

Feel free to go ahead with that

sbueringer commented 2 months ago

Doesn't hurt to start with a clean slate to reduce confusion :)

cahillsf commented 2 months ago

/close

in favor of https://github.com/kubernetes-sigs/cluster-api/issues/11133

k8s-ci-robot commented 2 months ago

@cahillsf: Closing this issue.

In response to [this](https://github.com/kubernetes-sigs/cluster-api/issues/9688#issuecomment-2327443686): >/close > >in favor of https://github.com/kubernetes-sigs/cluster-api/issues/11133 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.