all hooks are down. - Githubissues

reimertz commented 8 years ago

Error communicating with worker http://worker1:10000/marak/echo

The streaming connection errored in recieving data.

Please copy and paste this entire error message to Support.

Error: connect ECONNREFUSED
    at exports._errnoException (util.js:746:11)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1010:19)

reimertz commented 8 years ago

@marak

Marak commented 8 years ago

That is really odd.

I restarted all production instances and everything appears working.

It's probably been close to 4 months since I restarted the services, maybe it was a fluke. Also possible a user accidentally trigged an unknown error condition.

I'll investigate the server logs and see if I can figure out what happened.

Marak commented 8 years ago

I found the following error in the server logs, I suspect this is due to a missing try / catch around res.setHeader when a user attempts to set / write headers with invalid data.

_http_outgoing.js:333
    throw new Error('"name" and "value" are required for setHeader().');
    ^
Error: "name" and "value" are required for setHeader().
    at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:333:11)
    at new ClientRequest (_http_client.js:101:14)
    at Object.exports.request (http.js:49:10)
    at Object.exports.request (https.js:136:15)
    at Req._send (/root/hook.io/node_modules/hyperquest/index.js:136:21)
    at /root/hook.io/node_modules/hyperquest/index.js:50:21
    at process._tickCallback (node.js:355:11)

I hot-patched this previously for res.writeHead, I just now added the same patch for res.setHeader. Not sure if that was the issue, but it was causing some instances to crash.

Will continue to monitor the services.

reimertz commented 8 years ago

Awesome work! Thanks for fixing it so quickly.

Marak commented 8 years ago

Glad to help.

Sorry for the service disruption.

I'm not 100% convinced the outage was caused entirely by the uncaught error on res.setHeader. That was causing the process to crash, but the process should have instantly restarted, so hook services would be intermittent and not completely down. Possibly many of these uncaught errors stacked up and caused a spawn issue.

If you notice any additional down-time or unprocessed requests please let me know. Generally the system should perform very well.

stackvana / hook.io

all hooks are down. #190