HAProxy Restart coordination

[sttts]

I was wondering about the current logic during deployments in Marathon and I guess this ticket is about this topic. Some questions:

• During the time Marathon starts up new app instances, will they immediately appear in Bamboo's haproxy as servers?
• Will this wait until Marathon health checks are green?
• What happens with old servers? Will they just disappear from the haproxy and all existing connections will die?
• Will Marathon kill the app instances first and then tell Bamboo about it? Or does Bamboo will be able to drain those instances before Marathon will kill them?

[j1n6]

Hi @sttts, I see some very good questions that i've been thinking about improving. I will try to answer them:

• When Marathon starts up new app, they appear in Bamboo as soon as it gets notified by Marathon change notification callback. At beginning, new tasks are still under staging stage. To avoid directing traffic to staging or killed Marathon/Mesos tasks, Bamboo take the first Healthcheck from Marathon (it's configured by you) and put it into the HAProxy configuration. So we use HAProxy to identify liveness of the task. It only supports GET on HTTP at the moment. There are some drawbacks. Because Marathon supports multiple healthchecks while bamboo only takes account of first check. I wonder if Marathon provides a single endpoint to summarise N checks.
• Old servers will disappear from the Haproxy config. You might get packet drops if you have on going connections. To avoid dropping data packets, you want to look for configuring Linux to auto retry TCP connection.
• I think my first point somewhat answers your last question - if a running task doesn't pass HAProxy's healthcheck, it shouldn't route traffic to the dead tasks; I'm not sure the order of killing app in Marathon. (BUT I would love to know!).

[j1n6]

This ticket is about multiple Bamboo servers coordinate and running sequential rolling restart, rather than say 10 of Bamboo server reload at around the same time. I'm still considering if this feature I just described is necessary.

[sttts]

A related discussion in the Marathon project: https://github.com/mesosphere/marathon/issues/712

[xiaods]

@activars if the expected is accepted, why not use marathon to control the rolling update?

[j1n6]

Maybe that's one of the option. I thought about keep a lock in ZK and rolling update by themselves according to a preconfigured order. the zk will store these information:

• Checksum of the final configuration file: ensure all resulting configuration files per bamboo proxy is the same
• Who is working on the update and when it has finished for the next
• Liveness of Bamboo in zk per deployment node