QubitProducts / bamboo

HAProxy auto configuration and auto service discovery for Mesos Marathon
Apache License 2.0
794 stars 214 forks source link

How can I make bamboo update ACL automatically when restart the app? #101

Closed hyper0x closed 9 years ago

hyper0x commented 9 years ago

As it said in title. Thanks.

Vestaxis commented 9 years ago

+1. I'm running into this issue was well. I have to go into the Bamboo UI and edit and close (save is not necessary) the application before haproxy/bamboo picks up the restarted app.

j1n6 commented 9 years ago

This should be an easy fix. I will look into it.

On 30 Mar 2015, at 08:44, Vestaxis notifications@github.com wrote:

+1. I'm running into this issue was well. I have to go into the Bamboo UI and edit and close (save is not necessary) the application before haproxy/bamboo picks up the restarted app.

— Reply to this email directly or view it on GitHub https://github.com/QubitProducts/bamboo/issues/101#issuecomment-87581200.

hyper0x commented 9 years ago

@activars Great!

hyper0x commented 9 years ago

I found it accept a json-formatted response from [marathon_endpoint]/v2/tasks. But, the actual situation is, it will receive a plaintext after GET from the url. Is that the reason?

See: https://github.com/QubitProducts/bamboo/blob/master/services/marathon/marathon.go#L119

j1n6 commented 9 years ago

You have to set content type to application/json to get the json format.

hyper0x commented 9 years ago

@activars But the response is plaintext in Marathon 0.8.1.

j1n6 commented 9 years ago

I tested 0.8.1, so the default tasks API behaviour has changed.

As I remember with Marathon 0.6.* without "Accept: application/json", it returns tasks string with line breaks.

Marathon 0.7.* gives JSON response without specifying 'Accept: application/json'. I believe this tasks API behaviour has rollbacked to 0.6 in recent 0.8.1 release. I was afraid of those API behaviour changes, so added the http header in all marathon api request to enforce JSON request in previous release.

I think the line you quoted doesn't affect Bamboo:

$>  curl -H "Accept: application/json" http://10.141.141.10:8080/v2/tasks
{"tasks":[]}%

Did you enable the http_callback on Marathon?

hyper0x commented 9 years ago

I got it, and I tried it. As you said, it returned a json response with 'Accept: application/json'. Thank you for your answer. :+1:

btw, What about current issue?

hyper0x commented 9 years ago

@activars I will code to resolve this problem. But may be I start Marathon w/o its callback system.

hyper0x commented 9 years ago

@activars I enabled Marathon callback system, but the problem is still there.

I run (Bomboo can cover it):

curl -H "Content-Type:application/json" -d "" "http://<marathon_ip>:<marathon_port>/v2/eventSubscriptions?callbackUrl=<bamboo_ip>:<bamboo_port>/api/marathon/event_callback"

And get the result:

{"clientIp":"<marathon_ip>","callbackUrl":"<bamboo_ip>:<bamboo_port>/api/marathon/event_callback","eventType":"subscribe_event","timestamp":"2015-04-01T07:24:17.592Z"}

Is that right?

But no any additional output while I restart (or scale or new) the app in Marathon. On the other hand, I run follow cmd:

curl -d "{\"EventType\": \"fake event\"}" "http://<bamboo_ip>:<bamboo_port>/api/marathon/event_callback"

Then it seems everything is ok (response: Got it!).

I guess that is a question of Marathon about configuration.

j1n6 commented 9 years ago

I have a good solution for this. I would like to get chore #92 merged first and tackle this improvement next. Just being busy solving some internal issues, trying to work on this components in the evening.

bacoboy commented 9 years ago

:+1: Looking forward to the next release! Working around this is painful

j1n6 commented 9 years ago

I've worked out a solution. Please test master branch. Could you close the issue if the master branch solves the problem for you?

hyper0x commented 9 years ago

OK,I will try.

hyper0x commented 9 years ago

The problem will be resolved via reload haproxy on startup, right? What is the principle?

hyper0x commented 9 years ago

I resolved this problem, and submitted a pull request to marathon to resolve what it will interrupt the http event sending process if find one of the callback url's format is incorrect.

j1n6 commented 9 years ago

The PR you mentioned has been merged. The master branch now force an update on Bamboo start.

hyper0x commented 9 years ago

I konw, but I mean why do you say your solution(force an update on bamboo start) can make bamboo update acl automatically when the app/task in marathon is restarted? This feature has been ok already, isn't it? Do I make myself clear?

j1n6 commented 9 years ago

I think I was lost in the conversation... my bad. So there are three issues:

  1. Ensure the http callback indeed happens (Improved by your #105 PR)
  2. When bamboo started, in some situation, no events happening, the only way to force haproxy configuration changes is to change some record. (this is what i'm fixing, sorry if i misunderstood what you were asking) (Improved by #106 )
  3. When application restart in Marathon, there bamboo should discover the restart and ensure an update happen as quickly as possible (this is what you were asking)

It's likely marathon doesn't send an event when application restarted.

hyper0x commented 9 years ago

Great! That would be perfect. ;-)

As you guess, the event has not been sent. But I already got it. The problem is caused by a number of reasons(me & marathon: https://github.com/mesosphere/marathon/pull/1363)

j1n6 commented 9 years ago

Thanks. It looks like a problem upstream. I think it's better to solve the issue in Marathon.

On 4 Apr 2015, at 14:23, Robert Hao notifications@github.com wrote:

Great! That would be perfect. ;-)

As you guess, the event has not been sent. But I already got it. The problem is caused by a number of reasons(me & marathon: mesosphere/marathon#1363 https://github.com/mesosphere/marathon/pull/1363)

— Reply to this email directly or view it on GitHub https://github.com/QubitProducts/bamboo/issues/101#issuecomment-89577114.

hyper0x commented 9 years ago

I agree totally. I will test the newest revision of bamboo at tomorrow and close this issue. Thank you.

bacoboy commented 9 years ago

Can we get a 0.2.10 release cut for this?

j1n6 commented 9 years ago

@bacoboy do you mean a release for the 1 & 2 in the following quote (3 is likely to be improvement in Marathon upstream):

  1. Ensure the http callback indeed happens (Improved by your #105 PR)
  2. When bamboo started, in some situation, no events happening, the only way to force haproxy configuration changes is to change some record. (this is what i'm fixing, sorry if i misunderstood what you were asking) (Improved by #106 )
  3. When application restart in Marathon, there bamboo should discover the restart and ensure an update happen as quickly as possible (this is what you were asking)
bacoboy commented 9 years ago

I'm mostly interested in #106, but there hasn't been a release since Feb. So I'm asking for a more current official release. Thanks.

j1n6 commented 9 years ago

@bacoboy thanks for reminding me this. I will cut a release today.

j1n6 commented 9 years ago

Done, here's the release note: https://github.com/QubitProducts/bamboo/releases/tag/v0.2.10

bacoboy commented 9 years ago

Sweet. Thanks!