Closed vitosans closed 5 years ago
@vitosans The logs looks quite normal. Are you sure you're talking to the correct Marathon instance? http://marathon.mesos:8080
is the DC/OS system Marathon, that means that the app with the external
label must be installed in "Services", not in marathon-user
instance. Also consider adding HAPROXY_0_PORT
or HAPROXY_0_VHOST
label.
Hi @deric
I was just following the DC/OS tutorial here:
https://dcos.io/docs/1.10/networking/marathon-lb/marathon-lb-advanced-tutorial/
and running:
dcos package install marathon-lb
to set up both internal and external marathon-lb in the past for 1.9 this was a no issue. But right now I can reproduce it 100% of the time. the external refuses to start. I have even reinstalled the public-slave thinking I might have goofed on something. The internal has no issue at all.
I see this in the logs:
2017-10-23 18:08:17,075 marathon_lb: received event of type failed_health_check_event
2017-10-23 18:08:22,093 marathon_lb: received event of type failed_health_check_event
2017-10-23 18:08:27,115 marathon_lb: received event of type failed_health_check_event
2017-10-23 18:08:27,116 marathon_lb: received event of type unhealthy_instance_kill_event
2017-10-23 18:08:27,129 marathon_lb: received event of type status_update_event
2017-10-23 18:08:27,129 marathon_lb: received event of type instance_changed_event
2017-10-23 18:08:27,129 marathon_lb: fetching apps
but don't see what the failed event is. I have tried getting a shell on the container running the external to reproduce the failed health check. What concerns me is this is a fresh install nothing changed all done via the book except I have added portworx to test. I will try your suggestions and then read the marathon-lb.py and see if I can recreate the health check its failing.
I have the same issue here. The weird thing is that 1 marathon-lb
instance always has health-check green. But when I tried to scale it to 2 instances, the new one always failed with exactly the same error messages described in this issue.
When I was on DC/OS version 1.9 I never had this issue.
My DC/OS cluster is serving production traffic. It's never wracking to see production env running under 1 haproxy, which may be running to the same issue any minute!
I was having this same issue after our cluster suffered a network outage over the weekend. we have two public slaves, and on one of them marathon-lb would not start. We saw the following error in the task log by running:
dcos marathon task list --json /marathon-lb
message": "Task was killed since health check failed. Reason: BufferOverflowException: Exceeded configured max-open-requests value of [32]. This means that the request queue of this pool (HostConnectionPoolSetup(10.9.x.x,9090,ConnectionPoolSetup(ConnectionPool Settings(4,0,5,32,1,30 seconds,ClientConnectionSettings(Some(User-Agent: akka-http/10.0.6),10 seconds,1 minute,512,None,,List(),ParserSettings(2048,16,64,64,8192,64,268435456,256,1048576,Strict,RFC6265,true,Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, User-Agent -> 32),false,,,)),TCPTransport(None,ClientConnectionSettings(Some(User-Agent: akka-http/10.0.6),10 seconds,1 minute ,512,None,,List(),ParserSettings(2048,16,64,64,8192,64,268435456,256,1048576,Strict,RFC6265,true,Full,Error,Map(If-Range -> 0, If-Modified-Since -> 0, If-Unmodified-Since -> 0, default -> 12, Content-MD5 -> 0, Date -> 0, If-Match -> 0, If-None-Match -> 0, U ser-Agent -> 32),false,,,)))),akka.http.scaladsl.HttpConnectionContext$@549e393c,akka.event.MarkerLoggingAdapter@3a0c7578))) has completely filled up because the pool currently does not process requests fast enough to handle the incomi ng request load. Please retry the request later. See http://doc.akka.io/docs/akka-http/current/scala/http/client-side/pool-overflow.html for more information.",
was able to fix by forcing a leader election by running the following command on the leader node.
systemctl restart dcos-mesos-master
after a new leader was elected, marathon-lb started up and the health check passed successfully.
Hi, I'm facing a similar issue. It's healthy on one node, but unhealthy on the other. The above mentioned fix didn't work for me. Is there any other workaround?
Similar issues here during an upgrade from 1.9.4 to 1.10.4. Apparently solved only uninstalling and reinstalling marathon-lb, with much downtime.
@wobes what version of DC/OS are you using? It sounds like you had an issue with mesos overall, but I can't tell from your logs.
@pramodhkp what errors are you seeing in the Marathon-LB log?
@mimmus How many masters do you have, and how many public agents? What version of Marathon-LB are/were you on? Did you first upgrade to 1.9.7 per the DC/OS upgrade instructions? If you have 3+ masters, you should be able to upgrade in-place, and I wouldn't expect Marathon-LB to require a reinstall (at the most, it should require a restart, but it shouldn't even require that).
@justinrlee
@mimmus What errors and symptoms did you see when your Marathon-LB stopped working? Do you happen to have any logs?
Actually, if you were seeing Marathon-LB continually restarting, that's potentially caused by this: https://jira.mesosphere.com/browse/MARATHON-7572
The 'fix' is to switch from HTTP
to MESOS_HTTP
healthchecks. I have a hunch that this is what you saw.
@justinrlee I'm sorry to have saved no logs :( Marathon-lb stopped to work just after completing cluster upgrade. I had two orders of problems:
@justinrlee Mine was a false alarm. We had one more service running on one of the service ports that was causing an issue.
@justinrlee The version at the time we had this initial issue was 1.10.1, however we have also seen it with 1.10.3. The issue seems be be with the health check using connection pooling (akka-http). if you have access to the mesosphere support tickets see Request #8942.
Here is the response on that request:
"Update on the issue Marathon team acknowledged that connection pooling of HTTP healtchecks (that is causing this issue) is undesirable for the pattern of requests we are making. https://jira.mesosphere.com/browse/MARATHON-7940 has been filed to help address the connection pooling issue that happens when the socket fails to close.
We do not know which release of DC/OS would that change be implemented in, but for the time being we would recommend using MESOS_HTTP healthchecks instead of now deprecated marathon HTTP healthchecks."
so the work around is to modify the health check that the masters use to check the health of marathon-lb. Also like I mentioned before, forcing a leader election also cleared this issue for us.
This morning, serious problems: marathon-lb continously reloading, I'm unable ro recover. Suggested workarounds don't work.
I believe that this issue and https://github.com/mesosphere/marathon-lb/issues/504 refer to the same problem.
Please check my reply to https://github.com/mesosphere/marathon-lb/issues/504 [1]. I am going to close this issue. Let me know if there is anything else I can help with. Thanks!
[1] https://github.com/mesosphere/marathon-lb/issues/504#issuecomment-398883193
DCOS - v1.10 Marathon-LB v1.11.1
My external marathon-lb on my public agent seems to be suffering from this kind of related bug - This is on DCOS 1.10 when marathon-lb restarts it never recovers. At first I thought the issue was when an application was deployed with:
and the marathon-lb restarted due to this issue. Yesterday I was able to recover by terminating all apps that have external ports mapped, and then:
I was able to get marathon-lb back. But today it started happening and I am not able to recover no matter what I try. My log file looks like this:
So interesting things that stand out are:
and
and
This just started happening to me today, yesterday the symptoms where much differnet than today.