weyfonk
commented
5 months ago Additional QA
Problem
Correcting drift on Fleet-deployed resources would create a new Helm release, and a new sh.helm.<ID> secret every time, leading to an expanding set of stored secrets and Helm history items. This could lead to performance issues.
Solution
Helm Rollback operations, used internally by Fleet to correct drift, now obey Fleet's global limit on Helm history, restricting the number of kept history items to 2.
Testing
(See repro steps above)
-
Create a
GitRepowith drift correction enabled, either via the above example, or as follows:kind: GitRepo apiVersion: fleet.cattle.io/v1alpha1 metadata: name: test-drift-secrets spec: repo: https://github.com/rancher/fleet-test-data paths: - simple-chart correctDrift: enabled: true force: true -
Edit the deployment. In this
simple-chartexample, this could consist in editing theConfigMapcreated from thisGitRepo. -
Check that even after Fleet restores the deployment to its specified state (undoing manual changes), Helm history for the corresponding release still contains only 2 elements.
sbulage
This is a forwardport issue for #2515, automatically created via GitHub Actions workflow initiated by @aruiz14
Original issue body:
SURE-8550
Issue description:
When enabling Self Healing (drift detection) Fleet will generate a new secret every time drift is detected. To a point where it might exhaust Rancher. Fleet 0.9.4
Business impact:
For the customer Rancher went down due to too many secrets being cached
Troubleshooting steps:
Disabling self healing will clean the secrets
Repro steps:
helm historycommands in the target namespace and specifying the Helm release name.Workaround:
Is a workaround available and implemented? yes What is the workaround: disable self healing (disabling self healing also remove all the secrets)
Actual behavior:
self healing is not cleaning up the secrets
Expected behavior:
self-healing not to create so many secrets
Files, logs, traces:
Additional notes: