mozilla-services / megaphone

Firefox Global Broadcast API
Mozilla Public License 2.0
38 stars 12 forks source link

heartbeat hangs instead of failing #59

Closed glasserc closed 6 years ago

glasserc commented 6 years ago

On my machine, the __heartbeat__ endpoint works fine when the database is up, but when I shut it down, the same endpoint just hangs, with megaphone dumping errors to console like Error: Can't connect to MySQL server on 'golden-cloud.local' (111). After about 30 seconds, I get an error response like this:

*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8000 (#0)
> GET /__heartbeat__ HTTP/1.1
> Host: localhost:8000
> User-Agent: curl/7.61.0
> Accept: */*
> 
< HTTP/1.1 503 Service Unavailable
< Content-Type: text/html; charset=utf-8
< Server: Rocket
< Content-Length: 510
< Date: Mon, 24 Sep 2018 23:10:22 GMT
< 

            <!DOCTYPE html>
            <html>
            <head>
                <meta charset="utf-8">
                <title>503 Service Unavailable</title>
            </head>
            <body align="center">
                <div align="center">
                    <h1>503: Service Unavailable</h1>
                    <p>The server is currently unavailable.</p>
                    <hr />
                    <small>Rocket</small>
                </div>
            </body>
            </html>
* Connection #0 to host localhost left intact

Technically, I guess this is correct -- the 503 indicates that the heartbeat has failed -- but based on the structure of the JSON response, I assumed that it was possible for a response to have status be ok but database be error.

pjenvey commented 6 years ago

Most db errors render a JSON response, but this one's a failure to checkout a db conn from the pool. Due to our web framework (Rocket) this is done in a rocket request guard which currently lacks an easy way to customize error responses.

It can be worked around though.