search

jenkinsci / kubernetes-operator

Kubernetes native Jenkins Operator
https://jenkinsci.github.io/kubernetes-operator
Other
591 stars 231 forks source link

fix(backup): Duplicated backups & restores #1021

Closed DionJones615 closed 1 week ago

DionJones615 commented 2 weeks ago

Changes

Move bar.Client.Update() function beneath the bar.Client.Get() function to prevent a situation that causes the reconciliation to repeat itself at this point, causing two full restores to complete at every startup.

The same can be true of a backup if/when the operator restarts and bar.Client.Get() has not run yet.

Before Fix:

2024-06-14T21:25:03.409Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "jenkins"}
2024-06-14T21:25:03.409Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "jenkins"}
2024-06-14T21:25:03.435Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "jenkins"}
2024-06-14T21:25:03.457Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "jenkins"}
2024-06-14T21:25:03.484Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "jenkins"}
2024-06-14T21:25:03.484Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-06-14T21:25:03.484Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "jenkins"}
2024-06-14T21:25:03.484Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[eks.amazonaws.com/role-arn:arn:aws:iam::REDACTED:role/REDACTED]   {"cr": "jenkins"}
2024-06-14T21:25:03.643Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "jenkins"}
2024-06-14T21:25:03.643Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "jenkins"}
2024-06-14T21:25:03.649Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "jenkins"}
2024-06-14T21:25:03.655Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "jenkins"}
2024-06-14T21:25:03.655Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "jenkins"}
2024-06-14T21:25:03.655Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "jenkins"}
2024-06-14T21:25:03.655Z    DEBUG   controller-jenkins  Jenkins master pod is ready {"cr": "jenkins"}
2024-06-14T21:25:03.670Z    DEBUG   controller-jenkins  Jenkins API client set  {"cr": "jenkins"}
2024-06-14T21:25:03.683Z    DEBUG   controller-jenkins  Installed plugins 'REDACTED'    {"cr": "jenkins"}
2024-06-14T21:25:06.402Z    INFO    controller-jenkins  Restoring backup '597366'   {"cr": "jenkins"}
2024-06-14T22:16:03.885Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "jenkins"}
2024-06-14T22:16:03.885Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "jenkins"}
2024-06-14T22:16:03.912Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "jenkins"}
2024-06-14T22:16:03.924Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "jenkins"}
2024-06-14T22:16:03.942Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "jenkins"}
2024-06-14T22:16:03.942Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-06-14T22:16:03.942Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "jenkins"}
2024-06-14T22:16:03.942Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[eks.amazonaws.com/role-arn:arn:aws:iam::REDACTED:role/REDACTED]   {"cr": "jenkins"}
2024-06-14T22:16:04.085Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "jenkins"}
2024-06-14T22:16:04.085Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "jenkins"}
2024-06-14T22:16:04.090Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "jenkins"}
2024-06-14T22:16:04.100Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "jenkins"}
2024-06-14T22:16:04.100Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "jenkins"}
2024-06-14T22:16:04.100Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "jenkins"}
2024-06-14T22:16:04.100Z    DEBUG   controller-jenkins  Jenkins master pod is ready {"cr": "jenkins"}
2024-06-14T22:16:04.441Z    DEBUG   controller-jenkins  Jenkins API client set  {"cr": "jenkins"}
2024-06-14T22:16:04.609Z    DEBUG   controller-jenkins  Installed plugins 'REDACTED'    {"cr": "jenkins"}
2024-06-14T22:16:07.942Z    INFO    controller-jenkins  Restoring backup '597366'   {"cr": "jenkins"}

After Fix:

2024-06-20T16:46:53.264Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "jenkins"}
2024-06-20T16:46:53.265Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "jenkins"}
2024-06-20T16:46:53.281Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "jenkins"}
2024-06-20T16:46:53.294Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "jenkins"}
2024-06-20T16:46:53.316Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "jenkins"}
2024-06-20T16:46:53.316Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-06-20T16:46:53.316Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "jenkins"}
2024-06-20T16:46:53.316Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[eks.amazonaws.com/role-arn:REDACTED]  {"cr": "jenkins"}
2024-06-20T16:46:53.438Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "jenkins"}
2024-06-20T16:46:53.438Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "jenkins"}
2024-06-20T16:46:53.444Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "jenkins"}
2024-06-20T16:46:53.449Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "jenkins"}
2024-06-20T16:46:53.449Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "jenkins"}
2024-06-20T16:46:53.449Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "jenkins"}
2024-06-20T16:46:53.449Z    DEBUG   controller-jenkins  Jenkins master pod is ready {"cr": "jenkins"}
2024-06-20T16:46:53.464Z    DEBUG   controller-jenkins  Jenkins API client set  {"cr": "jenkins"}
2024-06-20T16:46:53.476Z    DEBUG   controller-jenkins  Installed plugins 'REDACTED'    {"cr": "jenkins"}
2024-06-20T16:46:54.106Z    INFO    controller-jenkins  Restoring backup '139533'   {"cr": "jenkins"}
2024-06-20T16:47:20.300Z    INFO    controller-jenkins  Waiting for Seed Job Agent `seed-job-agent`...  {"cr": "jenkins"}
2024-06-20T16:47:20.300Z    DEBUG   controller-jenkins  Reconciling Jenkins {"cr": "jenkins"}
2024-06-20T16:47:20.300Z    DEBUG   controller-jenkins  Operator credentials secret is present  {"cr": "jenkins"}
2024-06-20T16:47:20.323Z    DEBUG   controller-jenkins  Scripts config map is present   {"cr": "jenkins"}
2024-06-20T16:47:20.337Z    DEBUG   controller-jenkins  Init configuration config map is present    {"cr": "jenkins"}
2024-06-20T16:47:20.357Z    DEBUG   controller-jenkins  Base configuration config map is present    {"cr": "jenkins"}
2024-06-20T16:47:20.357Z    DEBUG   controller-jenkins  GroovyScripts Secret and ConfigMap added watched labels {"cr": "jenkins"}
2024-06-20T16:47:20.357Z    DEBUG   controller-jenkins  ConfigurationAsCode Secret and ConfigMap added watched labels   {"cr": "jenkins"}
2024-06-20T16:47:20.357Z    DEBUG   controller-jenkins  createServiceAccount with annotations map[eks.amazonaws.com/role-arn:REDACTED]  {"cr": "jenkins"}
2024-06-20T16:47:20.530Z    DEBUG   controller-jenkins  Service account, role and role binding are present  {"cr": "jenkins"}
2024-06-20T16:47:20.530Z    DEBUG   controller-jenkins  Extra role bindings are present {"cr": "jenkins"}
2024-06-20T16:47:20.541Z    DEBUG   controller-jenkins  Jenkins HTTP Service is present {"cr": "jenkins"}
2024-06-20T16:47:20.552Z    DEBUG   controller-jenkins  Jenkins slave Service is present    {"cr": "jenkins"}
2024-06-20T16:47:20.552Z    DEBUG   controller-jenkins  Kubernetes resources are present    {"cr": "jenkins"}
2024-06-20T16:47:20.552Z    DEBUG   controller-jenkins  Jenkins master pod is present   {"cr": "jenkins"}
2024-06-20T16:47:20.552Z    DEBUG   controller-jenkins  Jenkins master pod is ready {"cr": "jenkins"}
2024-06-20T16:47:20.745Z    DEBUG   controller-jenkins  Jenkins API client set  {"cr": "jenkins"}
2024-06-20T16:47:20.838Z    DEBUG   controller-jenkins  Installed plugins 'REDACTED'    {"cr": "jenkins"}
2024-06-20T16:47:20.839Z    DEBUG   controller-jenkins  Skipping restore backup, backup already restored    {"cr": "jenkins"}

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

See the contribution guide for more details.

Reviewer Notes

If API changes are included, additive changes must be approved by at least two OWNERS and backwards incompatible changes must be approved by more than 50% of the OWNERS.

Release Notes

- Fixed bug that caused duplicate backups and restores after jenkins-master or operator pod restart
brokenpip3 commented 2 weeks ago

thanks for your contribution!

Could you please merge the upstream master on your PR? I want to run the new tests with the backup image built on the fly and used before merging this PR.

About the multiple backup/restore issue I replied here: https://github.com/jenkinsci/kubernetes-operator/issues/962#issuecomment-2183917431