1-2 / 3 requests end in 503 Service Unavailable

[denisbetsi]

Have launched a cluster on amazon, 1) installed marathon 2) installed marathon-lb (1 instance ) default settings 3) launched an app, 1 instance with proxy label set to EXTERNAL

Everything goes green, when I attempt to load the page, at least 50% of all requests end with 503. There's no mention of this anywhere in the logs so I am not sure what the issue is. There's no load on the server, it's just me hitting refresh button.

Log files of marathon-lb has these warnings and alerts in it

Configuration file is valid
cat: /tmp/haproxy.pid: No such file or directory
[WARNING] 112/125741 (149) : Can't read first line of the server state file '/var/state/haproxy/global'
[WARNING] 112/125741 (149) : Can't read first line of the server state file '/var/state/haproxy/global'
[ALERT] 112/125741 (149) : sendmsg logger #1 failed: No such file or directory (errno=2)
[ALERT] 112/125741 (149) : sendmsg logger #2 failed: No such file or directory (errno=2)
[ALERT] 112/125741 (149) : sendmsg logger #1 failed: No such file or directory (errno=2)

I have attempted to delete marathon, start fresh and it always lands to the same issue.

It looks to be the HAProxy issue, it's just not clear what that is. There's only 1 instance so there should be no conflicts for it to now find a suitable server.

[lloesche]

That sounds like you have several frontends listening on the same port, some with backends some without backends, and haproxy is round robin balancing between those frontends. Check the generated config to confirm this.

Note that by default there are VHOST listeners on tcp/80 and tcp/443. You can disable those by specifying --dont-bind-http-https if you don't need them. So if you're running marathon-lb with default settings and then created a conflicting service that also listens on port 80 you would see the behaviour you described.

The ALERT log errors you're seeing are if you're running marathon-lb inside a Docker container because haproxy is unable to log to stdout/stderr. It requires a /dev/log device which the container doesn't have. You could mount one in from the host or install rsyslog inside the container, neither of which are particularly nice solutions imho.

[brndnmtthws]

Can you share the app definition you're using?

[denisbetsi]

It was an issue on my end with the docker file, was in the area of portMappings, setting it to

"portMappings": [ { "containerPort": 9000 } ]

and letting the rest of the system do it's thing.