search

gruntwork-io / health-checker

A simple HTTP server that will return 200 OK if all configured health checks pass.
MIT License
103 stars 36 forks source link

health-checker dies after fails to open any additional TCP connections after a period of time. #5

Closed josh-padnick closed 6 years ago

josh-padnick commented 6 years ago

I experienced some puzzling behavior where health-checker would run fine for a while, dutifully making its TCP checks. But then all TCP checks just started dieing and resulted in the following log output:

time="2018-02-05T00:15:50Z" level=info msg="Received inbound request. Beginning health checks..."
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8083 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8083 FAILED: dial tcp 0.0.0.0:8083: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8081 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8081 FAILED: dial tcp 0.0.0.0:8081: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="Attempting to connect to port 8082 via TCP..."
time="2018-02-05T00:15:50Z" level=warning msg="TCP connection to port 8082 FAILED: dial tcp 0.0.0.0:8082: socket: too many open files"
time="2018-02-05T00:15:50Z" level=info msg="At least one health check failed. Returning HTTP 504 response.\n"

The issue is that net.DialTimeout() should close its connection, but I accidentally omitted this .

josh-padnick commented 6 years ago

Resolved by #6.