Getting 503 when restarting application

[gabfssilva]

I'm getting the 503 status code when I'm trying to restart an application in Marathon. I have a question: the event from marathon to ZK is sent before or after the health check being healthy? It seems like the events are sent before the application is healthy. If it is a normal behaviour, I was wondering, is it possible to retry on 503 errors with haproxy?

Thanks in advance!

[j1n6]

Do you have health check configured in marathon app config? The healthchecks are used for HAProxy check. Details can be found in the reload templates.

On Jul 11, 2016, at 7:18 AM, Gabriel notifications@github.com wrote:

I'm getting the 503 status code when I'm trying to restart an application in Marathon. I have a question: the event from marathon to ZK is sent before or after the health check being healthy? It seems like the events are sent before the application is healthy. If it is a normal behaviour, I was wondering, is it possible to retry on 503 errors on haproxy?

Thanks in advance!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[gabfssilva]

Yes, I have configured it.

This is my app in marathon:

{
  "id": "/hello-java",
  "cmd": "java -jar hello-java-all* $HOST $PORT",
  "cpus": 0.1,
  "mem": 32,
  "disk": 20,
  "instances": 5,
  "executor": "//cmd",
  "healthChecks": [
    {
      "path": "/hello-java/health",
      "protocol": "HTTP",
      "portIndex": 0,
      "gracePeriodSeconds": 50,
      "intervalSeconds": 1,
      "timeoutSeconds": 1,
      "maxConsecutiveFailures": 0,
      "ignoreHttp1xx": false
    }
  ],
  "portDefinitions": [
    {
      "port": 10000,
      "protocol": "tcp",
      "labels": {}
    }
  ],
  "uris": [
    "https://github.com/gabfssilva/hello/blob/master/build/libs/hello-java-all.jar?raw=true"
  ],
  "container": null,
  "env": {}
}

Is there any place else where I need to configure something? It seems that bamboo tries to update the haproxy config on any event, even if the app is not healthy.

[j1n6]

Was app marked healthy on HAProxy health check admin UI?

On Jul 11, 2016, at 7:02 PM, Gabriel notifications@github.com wrote:

Yes, I have configured it.

This is my app in marathon:

{ "id": "/hello-java", "cmd": "java -jar hello-java-all* $HOST $PORT", "cpus": 0.1, "mem": 32, "disk": 20, "instances": 5, "executor": "//cmd", "healthChecks": [ { "path": "/hello-java/health", "protocol": "HTTP", "portIndex": 0, "gracePeriodSeconds": 50, "intervalSeconds": 1, "timeoutSeconds": 1, "maxConsecutiveFailures": 0, "ignoreHttp1xx": false } ], "portDefinitions": [ { "port": 10000, "protocol": "tcp", "labels": {} } ], "uris": [ "https://github.com/gabfssilva/hello/blob/master/build/libs/hello-java-all.jar?raw=true" ], "container": null, "env": {} } Is there any place else where I need to configure something? It seems that bamboo tries to update on any event, even if the app is not healthy.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[gabfssilva]

I just checked and sometimes on restart all the instances are "INI" and none L7OK/200. When I check marathon I always see at least one healthy instance.

[timoreimann]

There is a long standing issue about Bamboo adding tasks too early to HAProxy: #143.

The issue describes a few workarounds. We have come around enabling HAProxy's redispatch option to try a different backend on failure. This is also helpful when a container crashes unexpectedly.

I'd suggest to close this issue in favor of the one I referenced.