tasks sort order changing, causing unnecessary restarts

QubitProducts / bamboo

HAProxy auto configuration and auto service discovery for Mesos Marathon

Apache License 2.0

793 stars 214 forks source link

tasks sort order changing, causing unnecessary restarts #177

Closed rasputnik closed 8 years ago

rasputnik commented 8 years ago

We've been running bamboo 0.2.12 against marathon 0.7.5 happily for months, but today our haproxies have been reloading every minute or so - although the state of marathon is not changing.

Did some investigation (diffing haproxy.cfg, marathon and bamboo state URLs) it seems the order of tasks returned by http://our-marathon:8080/v2/tasks is changing - for reasons I can't understand.

This causes the generated haproxy.cfg to differ enough for bamboo to reload configuration.

our haproxy template is here: https://gist.github.com/rasputnik/3c4a846b5aedcef6ea02

Is there any way for me to sort the tasks (in the ' {{ range $page, $task := .Tasks }}' block) using bamboos templating?

Or am I going to have to patch our bamboo to ensure these tasks remain in order?

rasputnik commented 8 years ago

Odd, it looks like we already sort the taskList we get from marathon here:

https://github.com/QubitProducts/bamboo/blob/v0.2.12/services/marathon/marathon.go#L140

but for some reason it gets jumbled up again by the time it's merged into the main bamboo state.

Can anyone give me a clue what's going on here?

rasputnik commented 8 years ago

I've filed this ticket against marathon for some clarification of what should be expected from the rest api: https://github.com/mesosphere/marathon/issues/2567

rasputnik commented 8 years ago

Marathon line is that the tasks are returned in random order and it's consumers responsibility to sort them. I'm still digging around to find how the tasks become jumbled, since they're definitely sorted on line 140.

rasputnik commented 8 years ago

Liberal use of glog.Infof confirms the Tasks within the apps are not sorted when returned by createApps. The marathonTasks are sorted when passed to the function, I'm not seeing where they get jumbled.

Also appears that https://github.com/QubitProducts/bamboo/blob/v0.2.12/services/marathon/marathon.go#L235 is not sorting the Task list, just the parent Apps.

rasputnik commented 8 years ago

found 2 little bugs

sorting marathonTasks by .Id instead of .StagedAt resolves this consistently - unsure if there is an issue due to .StagedAt being a String so sorting inconsistently (our clocks recently changed so timezones may come into play), but this reliably fixes things for us
big chunks of marathon.go use []MarathonTask instead of MarathonTaskList, which may be causing sorts to happen oddly

PR incoming (just verifying the modified code handles normal deployments correctly).