RuntimeTools / appmetrics-elk

A connector that collects data using 'appmetrics' and sends it to ElasticSearch for use with Kibana
Apache License 2.0
29 stars 12 forks source link

unavailable ES URL crashes node #5

Closed benyamindfring closed 8 years ago

benyamindfring commented 8 years ago

Hi, when I input a wrong URL or block access on a real URL of ES, the whole node crashes due to the crash of this package.

if for example I configure the following, with a correct URL of a working ES, everything works great, but if for the sake of the test, I change the URL to something wrong, the node crashes with the error below.

working config:

var appMetricsConfig = {
hosts: [
    'http://es-cluster.domain.com'
],
index: 'appmetrics'
}
var appmetrics = require('appmetrics-elk').monitor(appMetricsConfig);

crashing config:

var appMetricsConfig = {
hosts: [
    'http://blabla.domain.com'
],
index: 'appmetrics'
}
var appmetrics = require('appmetrics-elk').monitor(appMetricsConfig);

this is the error from the node console logs:

appName-4 Elasticsearch WARNING: 2015-11-25T13:16:47Z
  Unable to revive connection: http://es-cluster-url.domain.com/

appName-4 Elasticsearch WARNING: 2015-11-25T13:16:47Z
  No living connections

PM2 App name:appName id:4 exited with code 1
PM2 Starting execution sequence in -fork mode- for app name:appName id:4
PM2 App name:appName id:4 online
appName-4 [Wed Nov 25 13:16:47 2015] com.ibm.diagnostics.healthcenter.loader INFO: Node Application Metrics 1.0.3.201511030958 (Agent Core 3.0.6.201511030958)
appName-4 [Wed Nov 25 13:16:47 2015] com.ibm.diagnostics.healthcenter.mqtt INFO: Connecting to broker localhost:1883
appName-4 Elasticsearch ERROR: 2015-11-25T13:16:48Z
  Error: Request error, retrying -- getaddrinfo ENOTFOUND
      at Log.error (/nodes/node_Stage_App_pm2/App_master_v1.0.10/node_modules/appmetrics-elk/node_modules/elasticsearch/src/lib/log.js:225:60)
      at checkRespForFailure (/nodes/node_Stage_App_pm2/App_master_v1.0.10/node_modules/appmetrics-elk/node_modules/elasticsearch/src/lib/transport.js:231:18)
      at HttpConnector.<anonymous> (/nodes/node_Stage_App_pm2/App_master_v1.0.10/node_modules/appmetrics-elk/node_modules/elasticsearch/src/lib/connectors/http.js:153:7)
      at ClientRequest.wrapper (/nodes/node_Stage_App_pm2/App_master_v1.0.10/node_modules/appmetrics-elk/node_modules/elasticsearch/node_modules/lodash/index.js:3095:19)
      at ClientRequest.EventEmitter.emit (events.js:95:17)
      at Socket.socketErrorListener (http.js:1547:9)
      at Socket.EventEmitter.emit (events.js:95:17)
      at net.js:830:16
      at process._tickDomainCallback [as _tickCallback] (node.js:459:13)

any idea how to fix it? many thanks

seabaylea commented 8 years ago

It should be relative straightforward to handle the error and produce a warning message instead - I'll take a look.

seabaylea commented 8 years ago

PR #10 covers this. You'll get a number of errors and warnings from Elasticsearch as the connection fails, but the application will continue to run.