search

travis-ci / cyclist

AWS ASG lifecycle thing :tada: :bicyclist:
MIT License
8 stars 2 forks source link

detach: detach instance from asg on termination to prevent head-of-line blocking #34

Closed igorwwwwwwwwwwwwwwwwwwww closed 6 years ago

igorwwwwwwwwwwwwwwwwwwww commented 7 years ago

refs travis-pro/dev#45

codecov-io commented 7 years ago

Codecov Report

Merging #34 into master will increase coverage by 1.33%. The diff coverage is 48.27%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #34      +/-   ##
==========================================
+ Coverage   62.57%   63.91%   +1.33%     
==========================================
  Files          11       11              
  Lines         994      859     -135     
==========================================
- Hits          622      549      -73     
+ Misses        330      266      -64     
- Partials       42       44       +2
Impacted Files Coverage Δ
cli.go 89.75% <100%> (+1.12%) :arrow_up:
server.go 67.81% <100%> (+1.46%) :arrow_up:
lifecycle.go 51.36% <25%> (-3.6%) :arrow_down:
lifecycle_event.go 100% <0%> (ø) :arrow_up:
package.go 100% <0%> (ø) :arrow_up:
sns.go 71.71% <0%> (+1.02%) :arrow_up:
token.go 36.84% <0%> (+1.12%) :arrow_up:
db.go 55.41% <0%> (+2.49%) :arrow_up:
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 9d93bd9...d71f1fb. Read the comment docs.

igorwwwwwwwwwwwwwwwwwwww commented 7 years ago

Looks like this approach has some issues:

time="2017-06-13T15:42:24Z" level=error msg="handling lifecycle transition failed" err="ValidationError: The instance i-fafafaf is not in InService or Standby.\n\tstatus code: 400, request id: fefefef instance=i-fafafaf method=POST path="/terminations/i-fafafaf"

The instance asg state is already Terminate:Wait by the time we are trying to remove it, at which point it is no longer possible to detach it.

I'm looking into performing the detachment earlier in the process, in the sns callback that handles the lifecycle actions.

soulshake commented 6 years ago

I think step scaling solved the problem this PR was meant to address. <3