mesosphere / marathon-lb

Marathon-lb is a service discovery & load balancing tool for DC/OS
Apache License 2.0
449 stars 301 forks source link

tasks are not passing health checks but tasks were receiving traffic #655

Closed mslee89 closed 4 years ago

mslee89 commented 4 years ago

Hello. I had an issue when I tried to deploy app on DC/OS. The new tasks are not passing health check, but those tasks were receiving traffic before health check query.

My info

App info (DC/OS) { "labels": { "HAPROXY_GROUP": "external", "HAPROXY_0_FRONTEND_HEAD": "", "HAPROXY_0_FRONTEND_BACKEND_GLUE": "", "HAPROXY_0_VHOST": "test.com" }, "id": "/test", "backoffFactor": 1.15, "backoffSeconds": 1, "container": { "portMappings": [ { "containerPort": 5000, "servicePort": 10131, "name": "python" } ], "type": "DOCKER", "volumes": [], "docker": { "image": "python:2", "forcePullImage": false, "privileged": false, "parameters": [] } }, "cpus": 2, "disk": 0, "healthChecks": [ { "gracePeriodSeconds": 300, "intervalSeconds": 60, "maxConsecutiveFailures": 3, "portIndex": 0, "timeoutSeconds": 60, "delaySeconds": 15, "protocol": "MESOS_HTTP", "path": "/health", "ipProtocol": "IPv4" } ], "instances": 3, "maxLaunchDelaySeconds": 3600, "mem": 2048, "gpus": 0, "networks": [ { "name": "dcos", "mode": "container" } ], "requirePorts": false, "upgradeStrategy": { "maximumOverCapacity": 0.1, "minimumHealthCapacity": 1 }, "killSelection": "YOUNGEST_FIRST", "unreachableStrategy": { "inactiveAfterSeconds": 0, "expungeAfterSeconds": 0 } }

Log

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
44.128.0.31 - - [04/Nov/2019 02:27:02] "GET / HTTP/1.1" 200 -
44.128.0.31 - - [04/Nov/2019 02:27:02] "GET / HTTP/1.1" 200 -

44.128.0.31 - - [04/Nov/2019 02:27:05] "GET / HTTP/1.1" 200 - **<< receiving service traffic before health check**
44.128.0.32 - - [04/Nov/2019 02:27:05] "GET /health HTTP/1.0" 200 -

I1104 11:27:12.801702 50743 checker_process.cpp:987] HTTP health check for task 'test-error.92419e2a-feaa-11e9-8b26-46ed464ae041' returned: 200
I1104 11:27:12.801801 50743 executor.cpp:435] Received task health update, healthy: true
mslee89 commented 4 years ago

Fixed : https://github.com/mesosphere/marathon-lb/releases/tag/v1.14.2