search

DataDog / extendeddaemonset

Kubernetes Extended Daemonset controller
Apache License 2.0
98 stars 13 forks source link

add parameter to configure max restarts for canary pause #51

Closed celenechang closed 3 years ago

celenechang commented 4 years ago

What does this PR do?

Currently the Canary autopause feature pauses the canary deployment if any of the canary pod containers restart more than 0 times. We have found that this is too sensitive, so this PR makes this a configurable parameter with a default value of 2. The maxRestarts parameter means that this is the number of restarts of a single container that is considered tolerable before we autopause.

It also adds the ability to disable the feature entirely via the spec.

Motivation

Better user experience

Additional Notes

A subsequent PR with a new parameter to define a stability (no restarts) duration requirement is planned.

Describe your test plan

Use an image such as bash:latest in the EDS configuration that causes the container to enter CLB. Increase the canary duration to be sufficiently large to give time for restarts.

Check if canary is automatically paused when the container restarts 3 times (default) without adding the autoPause.maxRestarts setting to the EDS spec.

Then try setting autoPause.maxRestarts to a larger number and see if the canary pauses after the corresponding number of restarts (see examples/foo-eds_v2.yaml for an example configuration).

Lastly, try setting autoPause.enabled to false and autoPause.maxRestarts to 0 and make sure that a restart does not pause the canary.

codecov-io commented 3 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@d72a1bc). Click here to learn what that means. The diff coverage is 46.66%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #51   +/-   ##
=========================================
  Coverage          ?   36.87%           
=========================================
  Files             ?       29           
  Lines             ?     1589           
  Branches          ?        0           
=========================================
  Hits              ?      586           
  Misses            ?      929           
  Partials          ?       74
Flag Coverage Δ
unittests 36.87% <46.66%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ers/extendeddaemonsetreplicaset/strategy/canary.go 0.00% <0.00%> (ø)
controllers/extendeddaemonset/controller.go 68.75% <42.85%> (ø)
pkg/controller/utils/pod/pod.go 16.43% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update d72a1bc...76d421b. Read the comment docs.