Add ERS canary NoRestartDuration and AutoFail configuration options to complement AutoPause behavior

[xornivore]

What does this PR do?

Adds new features to handle restarts in the canary phase:

• NoRestartsDuration for canary preventing end of the canary phase until there are no restarts within this time.
• AutoFail behaving similarly to pause but causing canary to be failed either on MaxRestarts or on exceeding MaxRestartsDuration for pod restarts.

Additional Notes

• Status.Conditions are used to record the last restart of an ERS pod and progressively track the difference between initial transition and update times.

• A future TODO - adding more pod termination statuses recorded as reason for canary pause or fail statuses, currently only CrashLoopBackOff and OOMKilled are captured, the rest is mapped to Unknown.

• There's quite a bit of refactoring of existing canary management logic to make it more testable with unit tests. Hoping reviewers can critique the changes and advise on possible improvements.

Describe your test plan

For NoRestartsDuration - configure a canary with a pod that is causing restarts within no restart duration, make sure that canary phase is not ended until this duration elapses.

For AutoFail - configure it with either MaxRestarts or MaxRestartsDuration and make sure the transition to canary-failed state takes place.

[codecov-io]

Codecov Report

Merging #66 (a84ce22) into master (4755dcd) will decrease coverage by 2.03%. The diff coverage is 11.93%.

@@            Coverage Diff             @@
##           master      #66      +/-   ##
==========================================
- Coverage   36.75%   34.71%   -2.04%     
==========================================
  Files          29       29              
  Lines        1570     1688     +118     
==========================================
+ Hits          577      586       +9     
- Misses        919     1028     +109     
  Partials       74       74              

Flag	Coverage Δ
unittests	34.71% <11.93%> (-2.04%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...trollers/extendeddaemonsetreplicaset/controller.go	53.81% <0.00%> (-0.99%)	:arrow_down:
...ers/extendeddaemonsetreplicaset/strategy/canary.go	0.00% <0.00%> (ø)
...endeddaemonsetreplicaset/strategy/rollingupdate.go	9.00% <0.00%> (ø)
...llers/extendeddaemonsetreplicaset/strategy/type.go	0.00% <ø> (ø)
...lers/extendeddaemonsetreplicaset/strategy/utils.go	16.85% <20.00%> (-2.38%)	:arrow_down:
pkg/controller/utils/pod/pod.go	15.78% <61.53%> (-0.65%)	:arrow_down:
controllers/extendeddaemonset/utils.go	94.87% <100.00%> (+1.53%)	:arrow_up:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 4755dcd...a84ce22. Read the comment docs.