search

jenkins-x / lighthouse

Apache License 2.0
183 stars 114 forks source link

fix: Retry foghorn LighthouseJob update using original variable instead of DeepCopy variable #1591

Closed dippynark closed 8 months ago

dippynark commented 8 months ago

We are using Lighthouse foghorn v1.14.4 and very occasionally it is getting into a continuous loop trying to update a LighthouseJob when there is a conflict.

Initially there is a failure with the following message:

Failed to update LighthouseJob status: Operation cannot be fulfilled on lighthousejobs.lighthouse.jenkins.io "[LIGHTHOUSE_JOB_NAME]": the object has been modified; please apply your changes to the latest version and try again

It does try again and succeeds: took 2 attempts to update Job

However, I think this is false positive because jobCopy (instead of &job) is being passed into retryModifyJob so on retry r.client.Get is overwriting jobCopy and reverting the updated status, meaning the retry is just updating the LighthouseJob to its current value.

This PR instead passes &job into retryModifyJob which matches with the Tekton controller: https://github.com/jenkins-x/lighthouse/blob/867b0b5c900ce2f500307e3de9bf1057972077e0/pkg/engines/tekton/controller.go#L146

I don't fully understand why foghorn is getting into a continuous loop, I would have thought the update just wouldn't happen (perhaps the noop update is actually triggering another event and foghorn is continuously conflicting with itself), but either way this should fix the problem because on next reconciliation we shouldn't get past the reflect.DeepEqual check.

For a future improvement, Lighthouse foghorn and other controllers could instead use RetryOnConflict from client-go to handle conflicts instead of the custom retryModifyJob function: https://pkg.go.dev/k8s.io/client-go/util/retry#RetryOnConflict

jenkins-x-bot commented 8 months ago

Hi @dippynark. Thanks for your PR.

I'm waiting for a jenkins-x member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository.
msvticket commented 8 months ago

/ok-to-test

msvticket commented 8 months ago

/approve

jenkins-x-bot commented 8 months ago

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: msvticket

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files: - ~~[OWNERS](https://github.com/jenkins-x/lighthouse/blob/main/OWNERS)~~ [msvticket] Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment
jenkins-x-bot commented 8 months ago

Failed to merge this PR due to:

failed merging [1591]: [Method Not Allowed]