Improve controller logic + fix tests flakiness

[clamoriniere]

What does this PR do?

• Add new CanaryFailed and CanaryPause conditions in ERS.status
• Add EDS.status.conditions with CanaryFailed and CanaryPause conditions.
• Remove EDS annotations update in the ERS reconcile loop.
• Check ERS.status.conditions in EDS reconcile loop to detec Canary paused or failed
• Update unit-tests and e2t tests according the logic update
• Add some reconcile loops documentations.
• In the ERS-controller, If the parent ExtendedDaemonset is not defaulted, we force the Defaulting by updating the ERS status with an error conditions. like this the EDS-controller received an event on an ERS update and requeue the parent EDS.
• Speed-up reconcile frequency in tests to avoid test flakiness
• Use only one Canary Pod in the canary strategy configuration in test.

Motivation

Remove test flakiness and improve EDS/ERS lifecycles.

Additional Notes

N/A

Describe your test plan

• Tests should not be flaky.

[codecov-io]

Codecov Report

Merging #74 (774e90d) into v0.5 (dffd446) will increase coverage by 1.66%. The diff coverage is 59.39%.

@@            Coverage Diff             @@
##             v0.5      #74      +/-   ##
==========================================
+ Coverage   31.84%   33.51%   +1.66%     
==========================================
  Files          41       41              
  Lines        2917     2987      +70     
==========================================
+ Hits          929     1001      +72     
  Misses       1899     1899              
+ Partials       89       87       -2     

Flag	Coverage Δ
unittests	33.51% <59.39%> (+1.66%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
api/v1alpha1/extendeddaemonset_default.go	55.69% <ø> (+1.37%)	:arrow_up:
api/v1alpha1/extendeddaemonset_types.go	100.00% <ø> (ø)
api/v1alpha1/extendeddaemonsetreplicaset_types.go	100.00% <ø> (ø)
api/v1alpha1/zz_generated.deepcopy.go	25.00% <0.00%> (-1.03%)	:arrow_down:
api/v1alpha1/zz_generated.openapi.go	0.00% <0.00%> (ø)
controllers/extendeddaemonset/metrics.go	0.63% <0.00%> (ø)
...endeddaemonsetreplicaset/strategy/rollingupdate.go	8.91% <0.00%> (-0.09%)	:arrow_down:
...ers/extendeddaemonsetreplicaset/strategy/canary.go	79.78% <46.15%> (+4.39%)	:arrow_up:
...trollers/extendeddaemonsetreplicaset/controller.go	56.11% <66.66%> (+2.13%)	:arrow_up:
...lers/extendeddaemonsetreplicaset/strategy/utils.go	53.27% <66.66%> (-1.87%)	:arrow_down:
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5d52e25...774e90d. Read the comment docs.

[clamoriniere]

@xornivore

Mind checking if we have this e2e test (I suspect we don't): working eds >> update eds >> (active canary with bad change) >> update with fix while canary active >> working eds

We don't, but it is a good idea. I will add this test.