search

ndelitski / rancher-alarms

Will kick your ass if found unhealthy service in Rancher environment
85 stars 20 forks source link

TypeError: Cannot read property 'toLowerCase' of undefined #23

Closed VAdamec closed 7 years ago

VAdamec commented 7 years ago 
docker run    -e RANCHER_ADDRESS=xxx.xxx.xxx.xxx:9000 \
  -e RANCHER_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxx  \
  -e RANCHER_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  \
  -e RANCHER_PROJECT_ID=1a5  \
  -e ALARM_SLACK_WEBHOOK_URL=http://xxx.xxx.xxx.xxx:8065/hooks/xxxxxxxxxxx      \
  --name rancher-alarms     ndelitski/rancher-alarms
> rancher-alarms@0.1.4 start /usr/src/app
> node bin/rancher-alarms.js

[INFO]   2017-1-4 11:55:55:232     composing config from env variables
[INFO]   2017-1-4 11:55:55:245     started with config:
{
    "rancher": {
        "address": "xxx.xxx.xxx.xxx:9000",
        "auth": {
            "accessKey": "xxxxxxxxxxxxxxxxxxxxxxxxx",
            "secretKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
        },
        "projectId": "1a5"
    },
    "pollServicesInterval": 60000,
    "notifications": {
        "*": {
            "targets": {
                "slack": {}
            },
            "healthcheck": {
                "pollInterval": 15000,
                "healthyThreshold": 3,
                "unhealthyThreshold": 4
            }
        }
    },
    "targets": {
        "slack": {
            "webhookUrl": "http://xxx.xxx.xxx.xxx:8065/hooks/xxxxxxxxxxxxxxxxxxxxxxxxx",
            "botName": "rancher-alarms"
        }
    }
}
[ERROR]  2017-1-4 11:55:55:597     [TypeError: Cannot read property 'toLowerCase' of undefined] TypeError: Cannot read property 'toLowerCase' of undefined
    at initServiceMonitor$ (/usr/src/app/src/server.es6:41:55)
    at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
    at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
    at invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37)
    at callInvokeWithMethodAndArg (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:172:16)
    at previousPromise (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:194:19)
    at new Promise (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:182:7)
    at AsyncIterator.enqueue (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:193:13)
    at AsyncIterator.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
    at Object.runtime.async (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:215:14)
    at initServiceMonitor (/usr/src/app/src/server.es6:51:22)
    at Array.map (native)
    at Object.callee$0$0$ (/usr/src/app/src/server.es6:25:39)
    at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
    at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)

npm ERR! Linux 3.10.0-327.28.3.el7.x86_64
npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
npm ERR! node v4.4.7
npm ERR! npm  v2.15.9
npm ERR! code ELIFECYCLE
npm ERR! rancher-alarms@0.1.4 start: `node bin/rancher-alarms.js`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the rancher-alarms@0.1.4 start script 'node bin/rancher-alarms.js'.
npm ERR! This is most likely a problem with the rancher-alarms package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node bin/rancher-alarms.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs rancher-alarms
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!
npm ERR!     npm owner ls rancher-alarms
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
ndelitski commented 7 years ago

Greetings @VAdamec! Which Rancher version do you use?

VAdamec commented 7 years ago

I'm using Docker image with latest tag, so version 1.3.0 I believe, I'll check it tomorrow.

courcelm commented 7 years ago

Having the same issue on Rancher 1.2.2

12/01/2017 08:54:07 12/01/2017 08:54:07> rancher-alarms@0.1.5 start /usr/src/app 12/01/2017 08:54:07> node bin/rancher-alarms.js 12/01/2017 08:54:07 12/01/2017 08:54:12[INFO] 2017-1-12 13:54:12:556 composing config from env variables 12/01/2017 08:54:12[INFO] 2017-1-12 13:54:12:569 started with config: 12/01/2017 08:54:12{ 12/01/2017 08:54:12 "rancher": { 12/01/2017 08:54:12 "address": "https://hostname", 12/01/2017 08:54:12 "auth": { 12/01/2017 08:54:12 "accessKey": "--", 12/01/2017 08:54:12 "secretKey": "--" 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "projectId": "1a5" 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "pollServicesInterval": 60000, 12/01/2017 08:54:12 "notifications": { 12/01/2017 08:54:12 "*": { 12/01/2017 08:54:12 "targets": { 12/01/2017 08:54:12 "email": { 12/01/2017 08:54:12 "recipients": [ 12/01/2017 08:54:12 "-@-.com" 12/01/2017 08:54:12 ], 12/01/2017 08:54:12 "subject": "[Rancher-Alarms] problems with #{serviceName}" 12/01/2017 08:54:12 } 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "healthcheck": { 12/01/2017 08:54:12 "pollInterval": 15000, 12/01/2017 08:54:12 "healthyThreshold": 3, 12/01/2017 08:54:12 "unhealthyThreshold": 4 12/01/2017 08:54:12 } 12/01/2017 08:54:12 } 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "targets": { 12/01/2017 08:54:12 "email": { 12/01/2017 08:54:12 "smtp": { 12/01/2017 08:54:12 "from": "Alarm of a Night Watch alarm@nightwatch.com", 12/01/2017 08:54:12 "auth": { 12/01/2017 08:54:12 "user": "-@-", 12/01/2017 08:54:12 "password": "-" 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "host": "smtp.gmail.com", 12/01/2017 08:54:12 "secureConnection": true, 12/01/2017 08:54:12 "port": "587" 12/01/2017 08:54:12 }, 12/01/2017 08:54:12 "template": "service #{serviceName} become #{monitorState}" 12/01/2017 08:54:12 } 12/01/2017 08:54:12 } 12/01/2017 08:54:12} 12/01/2017 08:54:13[ERROR] 2017-1-12 13:54:13:482 [TypeError: Cannot read property 'toLowerCase' of undefined] TypeError: Cannot read property 'toLowerCase' of undefined 12/01/2017 08:54:13 at initServiceMonitor$ (/usr/src/app/src/server.es6:41:55) 12/01/2017 08:54:13 at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) 12/01/2017 08:54:13 at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) 12/01/2017 08:54:13 at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) 12/01/2017 08:54:13 at invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37) 12/01/2017 08:54:13 at callInvokeWithMethodAndArg (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:172:16) 12/01/2017 08:54:13 at previousPromise (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:194:19) 12/01/2017 08:54:13 at new Promise (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:201:7) 12/01/2017 08:54:13 at AsyncIterator.enqueue (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:193:13) 12/01/2017 08:54:13 at AsyncIterator.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) 12/01/2017 08:54:13 at Object.runtime.async (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:215:14) 12/01/2017 08:54:13 at initServiceMonitor (/usr/src/app/src/server.es6:51:22) 12/01/2017 08:54:13 at Array.map (native) 12/01/2017 08:54:13 at Object.callee$0$0$ (/usr/src/app/src/server.es6:25:39) 12/01/2017 08:54:13 at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) 12/01/2017 08:54:13 at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) 12/01/2017 08:54:13 12/01/2017 08:54:13npm ERR! Linux 4.4.0-57-generic 12/01/2017 08:54:13npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start" 12/01/2017 08:54:13npm ERR! node v4.7.2 12/01/2017 08:54:13npm ERR! npm v2.15.11 12/01/2017 08:54:13npm ERR! code ELIFECYCLE 12/01/2017 08:54:13npm ERR! rancher-alarms@0.1.5 start: node bin/rancher-alarms.js 12/01/2017 08:54:13npm ERR! Exit status 1 12/01/2017 08:54:13npm ERR! 12/01/2017 08:54:13npm ERR! Failed at the rancher-alarms@0.1.5 start script 'node bin/rancher-alarms.js'. 12/01/2017 08:54:13npm ERR! This is most likely a problem with the rancher-alarms package, 12/01/2017 08:54:13npm ERR! not with npm itself. 12/01/2017 08:54:13npm ERR! Tell the author that this fails on your system: 12/01/2017 08:54:13npm ERR! node bin/rancher-alarms.js 12/01/2017 08:54:13npm ERR! You can get information on how to open an issue for this project with: 12/01/2017 08:54:13npm ERR! npm bugs rancher-alarms 12/01/2017 08:54:13npm ERR! Or if that isn't available, you can get their info via: 12/01/2017 08:54:13npm ERR! 12/01/2017 08:54:13npm ERR! npm owner ls rancher-alarms 12/01/2017 08:54:13npm ERR! There is likely additional logging output above. 12/01/2017 08:54:13 12/01/2017 08:54:13npm ERR! Please include the following file with any support request: 12/01/2017 08:54:13npm ERR! /usr/src/app/npm-debug.log

ndelitski commented 7 years ago

Seems the problem is in a new Rancher API format, will release a fix soon

fredleger commented 7 years ago

Running Rancher 1.1.4 seems to leads to the same error.

rancher-alarms_1 | 2017-01-26T21:48:56.583788330Z [INFO] 2017-1-26 21:48:56:583 discovered new running service, creating monitor for: undefined/letsencrypt rancher-alarms_1 | 2017-01-26T21:48:56.736443624Z [ERROR] 2017-1-26 21:48:56:729 [TypeError: Cannot read property 'toLowerCase' of undefined] TypeError: Cannot read property 'toLowerCase' of undefined rancher-alarms_1 | 2017-01-26T21:48:56.736554581Z at initServiceMonitor$ (/usr/src/app/src/server.es6:41:55) rancher-alarms_1 | 2017-01-26T21:48:56.736600232Z at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) rancher-alarms_1 | 2017-01-26T21:48:56.736642487Z at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) rancher-alarms_1 | 2017-01-26T21:48:56.736685244Z at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) rancher-alarms_1 | 2017-01-26T21:48:56.736725694Z at invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37) rancher-alarms_1 | 2017-01-26T21:48:56.736764915Z at callInvokeWithMethodAndArg (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:172:16) rancher-alarms_1 | 2017-01-26T21:48:56.736805032Z at previousPromise (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:194:19) rancher-alarms_1 | 2017-01-26T21:48:56.736844188Z at new Promise (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:201:7) rancher-alarms_1 | 2017-01-26T21:48:56.736882787Z at AsyncIterator.enqueue (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:193:13) rancher-alarms_1 | 2017-01-26T21:48:56.736922337Z at AsyncIterator.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) rancher-alarms_1 | 2017-01-26T21:48:56.736960976Z at Object.runtime.async (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:215:14) rancher-alarms_1 | 2017-01-26T21:48:56.737004393Z at initServiceMonitor (/usr/src/app/src/server.es6:51:22) rancher-alarms_1 | 2017-01-26T21:48:56.737042406Z at updateMonitors$ (/usr/src/app/src/server.es6:72:31) rancher-alarms_1 | 2017-01-26T21:48:56.737080311Z at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) rancher-alarms_1 | 2017-01-26T21:48:56.737119314Z at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) rancher-alarms_1 | 2017-01-26T21:48:56.737158924Z at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)

i really like the concept !

ndelitski commented 7 years ago

Sorry have no time to test just now, I will check this tomorrow with your Rancher version

On Fri, 27 Jan 2017 at 00:52, Frederic Leger notifications@github.com wrote:

Running Rancher 1.1.4 seems to leed to the same error.

''' rancher-alarms_1 | 2017-01-26T21:48:56.583788330Z [INFO] 2017-1-26 21:48:56:583 discovered new running service, creating monitor for: undefined/letsencrypt rancher-alarms_1 | 2017-01-26T21:48:56.736443624Z [ERROR] 2017-1-26 21:48:56:729 [TypeError: Cannot read property 'toLowerCase' of undefined] TypeError: Cannot read property 'toLowerCase' of undefined rancher-alarms_1 | 2017-01-26T21:48:56.736554581Z at initServiceMonitor$ (/usr/src/app/src/server.es6:41:55) rancher-alarms_1 | 2017-01-26T21:48:56.736600232Z at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) rancher-alarms_1 | 2017-01-26T21:48:56.736642487Z at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) rancher-alarms_1 | 2017-01-26T21:48:56.736685244Z at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) rancher-alarms_1 | 2017-01-26T21:48:56.736725694Z at invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37) rancher-alarms_1 | 2017-01-26T21:48:56.736764915Z at callInvokeWithMethodAndArg (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:172:16) rancher-alarms_1 | 2017-01-26T21:48:56.736805032Z at previousPromise (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:194:19) rancher-alarms_1 | 2017-01-26T21:48:56.736844188Z at new Promise (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:201:7) rancher-alarms_1 | 2017-01-26T21:48:56.736882787Z at AsyncIterator.enqueue (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:193:13) rancher-alarms_1 | 2017-01-26T21:48:56.736922337Z at AsyncIterator.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) rancher-alarms_1 | 2017-01-26T21:48:56.736960976Z at Object.runtime.async (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:215:14) rancher-alarms_1 | 2017-01-26T21:48:56.737004393Z at initServiceMonitor (/usr/src/app/src/server.es6:51:22) rancher-alarms_1 | 2017-01-26T21:48:56.737042406Z at updateMonitors$ (/usr/src/app/src/server.es6:72:31) rancher-alarms_1 | 2017-01-26T21:48:56.737080311Z at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40) rancher-alarms_1 | 2017-01-26T21:48:56.737119314Z at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22) rancher-alarms_1 | 2017-01-26T21:48:56.737158924Z at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21) '''

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ndelitski/rancher-alarms/issues/23#issuecomment-275525702, or mute the thread https://github.com/notifications/unsubscribe-auth/AC7ZWgMlEEJ4UJdEjpgdXIPuHNdf9Xtlks5rWRW0gaJpZM4Lajff .

knepperjm commented 7 years ago

I am seeing the same issue with v1.2.2

ndelitski commented 7 years ago

@VAdamec @courcelm @knepperjm @fredleger Hey guys! Please pull a new image rancher-alarms:v0.1.6 and verify that your problem is gone. Just filtered out system type services and all worked. Thanks!

fredleger commented 7 years ago

@ndelitski : at least i don't get the mentionned error in the logs.

But shouldn't it trigger a notification according to the logs bellow ? I didn't receive any slack notifications and i d'ont see any tries on the logs.. right ?

rancher-alarms_1 | 2017-01-28T12:58:32.709534897Z [INFO] 2017-1-28 12:58:32:709 discovered new running service, creating monitor for: openvpn-rancherlocal/openvpn-rancherlocal-server rancher-alarms_1 | 2017-01-28T12:58:32.712116043Z [INFO] 2017-1-28 12:58:32:711 new monitor up openvpn-rancherlocal/openvpn-rancherlocal-server: rancher-alarms_1 | 2017-01-28T12:58:32.712293235Z targets: "(SlackTarget {\"channel\":\"#monitoring\",\"botName\":\"rancher-webofmars\",\"template\":\"Salut webofmars! Your service #{serviceName} goes #{monitorState} (#{state}\"})" rancher-alarms_1 | 2017-01-28T12:58:32.712357310Z healthcheck: { rancher-alarms_1 | 2017-01-28T12:58:32.712399602Z "pollInterval": 15000, rancher-alarms_1 | 2017-01-28T12:58:32.712439895Z "healthyThreshold": 3, rancher-alarms_1 | 2017-01-28T12:58:32.712479622Z "unhealthyThreshold": 4 rancher-alarms_1 | 2017-01-28T12:58:32.712519457Z } rancher-alarms_1 | 2017-01-28T12:58:32.712557302Z rancher-alarms_1 | 2017-01-28T12:58:32.713634655Z [INFO] 2017-1-28 12:58:32:712 start polling openvpn-rancherlocal/openvpn-rancherlocal-server rancher-alarms_1 | 2017-01-28T13:00:18.515594317Z [INFO] 2017-1-28 13:0:18:515 service openvpn-rancherlocal/openvpn-rancherlocal-server active -> degraded rancher-alarms_1 | 2017-01-28T13:00:34.342592857Z [INFO] 2017-1-28 13:0:34:341 service openvpn-rancherlocal/openvpn-rancherlocal-server degraded -> active

fredleger commented 7 years ago

OK made more testing and the error is still here...

Steps to reproduce : a) rancher server v1.1.4 (possibilly more versions) b) start a letsencrypt container from catalog, not providing the provider credentials. That will end with a container unable to go to the healthy state c) see what happens in rancher-alarms logs

rancher-alarms_1 | 2017-01-28T13:14:02.655755334Z [INFO]   2017-1-28 13:14:2:655     discovered new running service, creating monitor for: undefined/letsencrypt
rancher-alarms_1 | 2017-01-28T13:14:02.794914466Z [ERROR]  2017-1-28 13:14:2:788     [TypeError: Cannot read property 'toLowerCase' of undefined] TypeError: Cannot read property 'toLowerCase' of undefined
rancher-alarms_1 | 2017-01-28T13:14:02.795038458Z     at initServiceMonitor$ (/usr/src/app/src/server.es6:47:55)
rancher-alarms_1 | 2017-01-28T13:14:02.795093818Z     at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
rancher-alarms_1 | 2017-01-28T13:14:02.795139459Z     at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
rancher-alarms_1 | 2017-01-28T13:14:02.795184750Z     at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
rancher-alarms_1 | 2017-01-28T13:14:02.795229187Z     at invoke (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:136:37)
rancher-alarms_1 | 2017-01-28T13:14:02.795311974Z     at callInvokeWithMethodAndArg (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:172:16)
rancher-alarms_1 | 2017-01-28T13:14:02.795362291Z     at previousPromise (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:194:19)
rancher-alarms_1 | 2017-01-28T13:14:02.795403353Z     at new Promise (/usr/src/app/node_modules/babel-core/node_modules/core-js/modules/es6.promise.js:201:7)
rancher-alarms_1 | 2017-01-28T13:14:02.795442319Z     at AsyncIterator.enqueue (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:193:13)
rancher-alarms_1 | 2017-01-28T13:14:02.795481198Z     at AsyncIterator.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
rancher-alarms_1 | 2017-01-28T13:14:02.795520137Z     at Object.runtime.async (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:215:14)
rancher-alarms_1 | 2017-01-28T13:14:02.795557903Z     at initServiceMonitor (/usr/src/app/src/server.es6:57:22)
rancher-alarms_1 | 2017-01-28T13:14:02.795593415Z     at updateMonitors$ (/usr/src/app/src/server.es6:78:31)
rancher-alarms_1 | 2017-01-28T13:14:02.795631120Z     at tryCatch (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:61:40)
rancher-alarms_1 | 2017-01-28T13:14:02.795674950Z     at GeneratorFunctionPrototype.invoke [as _invoke] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:328:22)
rancher-alarms_1 | 2017-01-28T13:14:02.795712584Z     at GeneratorFunctionPrototype.prototype.(anonymous function) [as next] (/usr/src/app/node_modules/babel-core/node_modules/regenerator/runtime.js:94:21)
rancher-alarms_1 | 2017-01-28T13:14:02.866913516Z
rancher-alarms_1 | 2017-01-28T13:14:02.906240563Z npm ERR! Linux 3.16.0-4-amd64
rancher-alarms_1 | 2017-01-28T13:14:02.908083387Z npm ERR! argv "/usr/bin/node" "/usr/bin/npm" "start"
rancher-alarms_1 | 2017-01-28T13:14:02.909088966Z npm ERR! node v4.7.2
rancher-alarms_1 | 2017-01-28T13:14:02.910406649Z npm ERR! npm  v2.15.11
rancher-alarms_1 | 2017-01-28T13:14:02.911955643Z npm ERR! code ELIFECYCLE
rancher-alarms_1 | 2017-01-28T13:14:02.914036469Z npm ERR! rancher-alarms@0.1.6 start: `node bin/rancher-alarms.js`
rancher-alarms_1 | 2017-01-28T13:14:02.915037964Z npm ERR! Exit status 1
rancher-alarms_1 | 2017-01-28T13:14:02.916216120Z npm ERR!
rancher-alarms_1 | 2017-01-28T13:14:02.917391046Z npm ERR! Failed at the rancher-alarms@0.1.6 start script 'node bin/rancher-alarms.js'.
rancher-alarms_1 | 2017-01-28T13:14:02.918779528Z npm ERR! This is most likely a problem with the rancher-alarms package,
rancher-alarms_1 | 2017-01-28T13:14:02.918925618Z npm ERR! not with npm itself.
rancher-alarms_1 | 2017-01-28T13:14:02.919798484Z npm ERR! Tell the author that this fails on your system:
rancher-alarms_1 | 2017-01-28T13:14:02.922091000Z npm ERR!     node bin/rancher-alarms.js
rancher-alarms_1 | 2017-01-28T13:14:02.922204841Z npm ERR! You can get information on how to open an issue for this project with:
rancher-alarms_1 | 2017-01-28T13:14:02.922244508Z npm ERR!     npm bugs rancher-alarms
rancher-alarms_1 | 2017-01-28T13:14:02.923549058Z npm ERR! Or if that isn't available, you can get their info via:
rancher-alarms_1 | 2017-01-28T13:14:02.923663754Z npm ERR!
rancher-alarms_1 | 2017-01-28T13:14:02.924818780Z npm ERR!     npm owner ls rancher-alarms
rancher-alarms_1 | 2017-01-28T13:14:02.924999328Z npm ERR! There is likely additional logging output above.
rancher-alarms_1 | 2017-01-28T13:14:02.942824249Z
rancher-alarms_1 | 2017-01-28T13:14:02.944711975Z npm ERR! Please include the following file with any support request:
rancher-alarms_1 | 2017-01-28T13:14:02.944842998Z npm ERR!     /usr/src/app/npm-debug.log

API relevant part :

{
"id": "1s182",
"type": "service",
"links": {
"self": "…/v1/projects/1a5/services/1s182",
"account": "…/v1/projects/1a5/services/1s182/account",
"consumedbyservices": "…/v1/projects/1a5/services/1s182/consumedbyservices",
"consumedservices": "…/v1/projects/1a5/services/1s182/consumedservices",
"environment": "…/v1/projects/1a5/services/1s182/environment",
"instances": "…/v1/projects/1a5/services/1s182/instances",
"serviceExposeMaps": "…/v1/projects/1a5/services/1s182/serviceexposemaps",
"containerStats": "…/v1/projects/1a5/services/1s182/containerstats",
},
"actions": {
"update": "…/v1/projects/1a5/services/1s182/?action=update",
"remove": "…/v1/projects/1a5/services/1s182/?action=remove",
"setservicelinks": "…/v1/projects/1a5/services/1s182/?action=setservicelinks",
"removeservicelink": "…/v1/projects/1a5/services/1s182/?action=removeservicelink",
"addservicelink": "…/v1/projects/1a5/services/1s182/?action=addservicelink",
"deactivate": "…/v1/projects/1a5/services/1s182/?action=deactivate",
},
"name": "letsencrypt",
"state": "updating-active",
"accountId": "1a5",
"assignServiceIpAddress": false,
"createIndex": 1,
"created": "2017-01-28T13:12:59Z",
"createdTS": 1485609179000,
"currentScale": 1,
"description": null,
"environmentId": "1e83",
"externalId": null,
"fqdn": null,
"healthState": "unhealthy",
"kind": "service",
"launchConfig": {
"dataVolumes": [
"/var/apps/pki:/etc/letsencrypt/production/certs",
],
"environment": {
"API_VERSION": "Production",
"AWS_ACCESS_KEY": "",
"AWS_SECRET_KEY": "",
"CERT_NAME": "test1.services.webofmars.com",
"CLOUDFLARE_EMAIL": "",
"CLOUDFLARE_KEY": "",
"DNSIMPLE_EMAIL": "",
"DNSIMPLE_KEY": "",
"DOMAINS": "test1.services.webofmars.com",
"DO_ACCESS_TOKEN": "",
"DYN_CUSTOMER_NAME": "",
"DYN_PASSWORD": "",
"DYN_USER_NAME": "",
"EMAIL": "contact@webofmars.com",
"EULA": "Yes",
"PROVIDER": "Route53",
"PUBLIC_KEY_TYPE": "RSA-2048",
"RENEWAL_TIME": "12",
},
"imageUuid": "docker:janeczku/rancher-letsencrypt:v0.3.0",
"kind": "container",
"labels": {
"io.rancher.container.agent.role": "environment",
"io.rancher.container.create_agent": "true",
"io.rancher.service.hash": "9dd582df1e7f3c53044802e521947bbe98a7d20f",
},
"logConfig": { },
"networkMode": "managed",
"privileged": false,
"publishAllPorts": false,
"readOnly": false,
"startOnCreate": true,
"stdinOpen": false,
"tty": false,
"version": "0",
"vcpu": 1,
},
"metadata": {
"io.rancher.service.hash": "b7439c5b25b6debb6f2a919275ee975edd911f97",
},
"publicEndpoints": null,
"removed": null,
"retainIp": null,
"scale": 1,
"scalePolicy": null,
"secondaryLaunchConfigs": [ ],
"selectorContainer": null,
"selectorLink": null,
"startOnCreate": false,
"transitioning": "yes",
"transitioningMessage": "Reconciling",
"transitioningProgress": null,
"upgrade": null,
"uuid": "0a7ef95e-2f80-46d5-ba2b-0ec0614ca848",
"vip": null,
},
ndelitski commented 7 years ago

Thank you @fredleger! Looks like you found another corner case, when the service was added after alarms container has been started

ndelitski commented 7 years ago

@fredleger Please check against rancher-alarms:v0.1.7 image

fredleger commented 7 years ago

@ndelitski : Good work ! This one work like a charm so far.

ndelitski commented 7 years ago

closing the issue, if anyone will get an error just paste it here again

VAdamec commented 7 years ago

Thanks, works perfectly

courcelm commented 7 years ago

Works well also on Rancher 1.3.3