Enable support for marathon task_killing state

[thiagoit]

With the current bamboo version, if you have a number of tasks running on marathon and you scale them down, the tasks will be killed and while bamboo is processing marathon event and updating haproxy configuration users will experience 503 errors.

Marathon 1.3 has implemented 2 features for graceful shutdown as shown below that can be used so that bamboo is aware that a task will be killed and updates haproxy to stop routing connections to it

taskKillGracePeriod: is the time between a task get a SIGTERM and waits for the task to die until a SIGKILL is send. The task is responsible for terminating in that time, until it is killed forcefully. TASK_KILLING: a new task state has been introduced in Mesos, which signals, that a task got a kill request and is in the grace period. Other tools (e.g. load balancer, service discovery etc) should not route traffic to tasks in that state

[timoreimann]

We recently implemented this in our fork of Bamboo by means of exposing the task state and changing the HAProxy configuration template file to only include a server if the associated task's state is not TASK_KILLING; i.e.,

{{- range $index, $task := .Tasks }}
{{- if not ( eq .State "TASK_KILLING" ) }}
        server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }}{{ if $app.HealthCheckPath }} check inter 30000{{ end }}{{ if eq $app.HealthCheckProtocol "HTTPS" }} ssl verify none{{ end }}
{{- end }}{{/* if not eq .State */}}
{{- end }}{{/* range tasks */}}

I suppose this could also be implemented in Go code only. For us, it was simple enough to just expose the state and make a minor modification to the template file.

Happy to upstream if that's desirable.

[j1n6]

We can add this to the default template. Can you make a PR?

On Apr 1, 2017, at 22:52, Timo Reimann notifications@github.com wrote:

We recently implemented this in our fork of Bamboo by means of exposing the task state and changing the HAProxy configuration template file to only include a server if the associated task's state is not TASK_KILLING; i.e.,

{{- range $index, $task := .Tasks }} {{- if not ( eq .State "TASK_KILLING" ) }} server {{ $app.EscapedId}}-{{ $task.Host }}-{{ $task.Port }} {{ $task.Host }}:{{ $task.Port }}{{ if $app.HealthCheckPath }} check inter 30000{{ end }}{{ if eq $app.HealthCheckProtocol "HTTPS" }} ssl verify none{{ end }} {{- end }}{{/ if not eq .State /}} {{- end }}{{/ range tasks /}} I suppose this could also be implemented in Go code only. For us, it was simple enough to just expose the state and make a minor modification to the template file.

Happy to upstream if that's desirable.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.