ndelitski / rancher-alarms

Will kick your ass if found unhealthy service in Rancher environment
85 stars 20 forks source link

Handle non-200 exceptions #39

Open flaccid opened 6 years ago

flaccid commented 6 years ago

The process/container will exit when something like the following is hit. It would be good to add in some exception handling to perhaps retry or simply log and ignore for a period.

It looks like every now and then polling to rancher server might have a transient issue, so it would be good to handle this.

[ERROR]  2017-10-30 23:53:17:159   [Error: RancherClientError: non-200 code response {
    "code": "ECONNRESET"
}] Error: RancherClientError: non-200 code response {
    "code": "ECONNRESET"
}
    at RancherClient._request$ (/usr/src/app/src/rancher.es6:50:13)
    at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as throw] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
    at GeneratorFunctionPrototype.invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37)
    at run (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:108:47)
    at /usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:119:28
    at flush (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/$.microtask.js:19:5)
    at nextTickCallbackWith0Args (node.js:489:9)
    at process._tickCallback (node.js:418:13)
npm ERR! Linux 4.4.0-59-generic
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
npm ERR! node v4.8.4
npm ERR! npm  v2.15.12
npm ERR! code ELIFECYCLE
npm ERR! rancher-alarms@0.1.8 start: `node bin/rancher-alarms.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the rancher-alarms@0.1.8 start script 'node bin/rancher-alarms.js'.
npm ERR! This is most likely a problem with the rancher-alarms package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node bin/rancher-alarms.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs rancher-alarms
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls rancher-alarms
npm ERR! There is likely additional logging output above.
npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
flaccid commented 6 years ago

In https://github.com/ndelitski/rancher-alarms/blob/master/src/rancher.es6#L36-L52:

    catch (resp) {
      throw new Error('RancherClientError: non-200 code response ' + JSON.stringify(resp, null, 4));
    }

The exception is caught, but instead of throwing can we retry or something?

Something to be more resilient and prevent rancher-alarms from exiting when it doesn't need to in this case. FYI @ndelitski