search

rcbops / rpc-maas

Ansible playbooks for deploying Rackspace Monitoring-as-a-Service within Openstack Environments
Apache License 2.0
32 stars 68 forks source link

v10.1.18: RackspaceMonitoringValidationError - alarmParseError - Failed to parse alarm #228

Closed byronmccollum closed 8 years ago

byronmccollum commented 8 years ago

Started seeing this in 10.1.18...

PLAY [rabbit] ***************************************************************** 

GATHERING FACTS *************************************************************** 
ok: [bc08a2dd-infra1_rabbit_mq_container-1ae54342]
ok: [bc08a2dd-infra1_rabbit_mq_container-96c29a24]
ok: [bc08a2dd-infra1_rabbit_mq_container-ff443063]

TASK: [maas_local | Get entity ID for physical_host] ************************** 
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24]
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342]
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063]

TASK: [maas_local | Validate if check exists] ********************************* 
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342]
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24]
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063]

TASK: [maas_local | Create check if it does not exist] ************************ 
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24]
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063]
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342]

TASK: [maas_local | Get check ID for newly created check] ********************* 
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342]
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063]
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24]

TASK: [maas_local | Validate if alarm exists] ********************************* 
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'})
failed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "delta": "0:00:01.997031", "end": "2015-12-11 00:48:03.280964", "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.283933"}
...ignoring
failed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item={'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "delta": "0:00:01.224140", "end": "2015-12-11 00:48:04.636171", "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:03.412031"}
...ignoring
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'})
failed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "delta": "0:00:01.312386", "end": "2015-12-11 00:48:02.488470", "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.176084"}
...ignoring
failed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item={'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "delta": "0:00:02.362091", "end": "2015-12-11 00:48:05.053890", "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:02.691799"}
...ignoring
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'})
changed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'})
failed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "delta": "0:00:02.497601", "end": "2015-12-11 00:48:03.865509", "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.367908"}
...ignoring
failed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item={'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}) => {"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "delta": "0:00:01.161495", "end": "2015-12-11 00:48:05.226314", "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:04.064819"}
...ignoring

TASK: [maas_local | Create alarm if it does not exist] ************************ 
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:54.141326', u'stdout': u'<Alarm: id=almyno8aeP, label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.458702', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:52.682624'}, {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:55.601866', u'stdout': u'<Alarm: id=alWybK5fT1, label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.271923', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:54.329943'}, {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:56.962498', u'stdout': u'<Alarm: id=alzVUuAW9l, label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}, u'stderr': u'', u'delta': u'0:00:01.156999', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:55.805499'}, {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:58.314845', u'stdout': u'<Alarm: id=alnYrpvCD5, label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is rskipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:54.020611', u'stdout': u'<Alarm: id=alwbgmADAZ, label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.329205', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:52.691406'}, {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:55.668497', u'stdout': u'<Alarm: id=alWHhf9XDY, label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.448378', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:54.220119'}, {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:56.894737', u'stdout': u'<Alarm: id=als1pNQON1, label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}, u'stderr': u'', u'delta': u'0:00:01.019669', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:55.875068'}, {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:58.386307', u'stdout': u'<Alarm: id=alt07U3NNd, label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is rskipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:53.821423', u'stdout': u'<Alarm: id=alwAoxtYGI, label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.109477', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_disk_free_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:52.711946'}, {'name': 'rabbitmq_disk_free_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_disk_free_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_disk_free_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:55.211417', u'stdout': u'<Alarm: id=aljWHZh48s, label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}, u'stderr': u'', u'delta': u'0:00:01.190727', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_mem_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:54.020690'}, {'name': 'rabbitmq_mem_alarm_status', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_mem_alarm_status"] != 1) { return new AlarmStatus(CRITICAL, "rabbitmq_mem_alarm_status triggered"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:56.448384', u'stdout': u'<Alarm: id=alWyQmKF5R, label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}, u'stderr': u'', u'delta': u'0:00:01.129503', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_max_channels_per_conn--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:55.318881'}, {'name': 'rabbitmq_max_channels_per_conn', 'criteria': u':set consecutiveCount=3 if (metric["rabbitmq_max_channels_per_conn"] > 10) { return new AlarmStatus(CRITICAL, "Detected RabbitMQ connections with > 10 channels, check RabbitMQ and all Openstack consumers"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:57.884949', u'stdout': u'<Alarm: id=alze8sZrav, label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.150314', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:57.164531'}, {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:59.667935', u'stdout': u'<Alarm: id=alcztrQxuG, label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.198971', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:58.468964'}, {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:01.156448', u'stdout': u'<Alarm: id=ala5rzAHJH, label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 0, 'item': {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.289177', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:47:59.867271'}, {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}])
failed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:03.865509', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 1, 'item': {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:02.497601', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:48:01.367908'}, {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chJs9elVwM --notification-plan npTechnicalContactsEmail --label rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24 --criteria ' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }'", "delta": "0:00:00.956843", "end": "2015-12-11 00:48:06.560366", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "delta": "0:00:02.497601", "end": "2015-12-11 00:48:03.865509", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "module_name": "shell"}, "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.367908", "stderr": "", "stdout": ""}, {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:05.603523"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 23, 'error_line': 1, 'error_position': 23, 'input': ' :set conseaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.296015', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:57.090292'}, {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:59.643110', u'stdout': u'<Alarm: id=alvTyr5Gh8, label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.049402', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:58.593708'}, {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:01.078499', u'stdout': u'<Alarm: id=aly4sAl5ot, label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 0, 'item': {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.231186', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:47:59.847313'}, {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}])
failed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:03.280964', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 1, 'item': {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:01.997031', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:48:01.283933'}, {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chfYGkCra9 --notification-plan npTechnicalContactsEmail --label rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063 --criteria ' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }'", "delta": "0:00:01.096529", "end": "2015-12-11 00:48:06.738768", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "delta": "0:00:01.997031", "end": "2015-12-11 00:48:03.280964", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "module_name": "shell"}, "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.283933", "stderr": "", "stdout": ""}, {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:05.642239"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 23, 'error_line': 1, 'error_position': 23, 'input': ' :set conseaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.242282', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_fd_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:56.642667'}, {'name': 'rabbitmq_fd_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_fd_used"],metric["rabbitmq_fd_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ file descriptors is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:47:59.504148', u'stdout': u'<Alarm: id=alpy8gceXK, label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.414871', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_proc_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:58.089277'}, {'name': 'rabbitmq_proc_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_proc_used"],metric["rabbitmq_proc_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ processes is reaching configured limit"); }'}])
skipping: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:00.966621', u'stdout': u'<Alarm: id=al6WtZDa8J, label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342 ...>', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 0, 'item': {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}, u'stderr': u'', u'delta': u'0:00:01.255064', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_socket_used_alarm_status--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:47:59.711557'}, {'name': 'rabbitmq_socket_used_alarm_status', 'criteria': u':set consecutiveCount=3 if (percentage(metric["rabbitmq_socket_used"],metric["rabbitmq_socket_total"]) >= 90 ) { return new AlarmStatus(CRITICAL, "RabbitMQ sockets is reaching configured limit"); }'}])
failed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:02.488470', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 1, 'item': {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:01.312386', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:48:01.176084'}, {'name': 'rabbitmq_msgs_excl_notifications', 'criteria': u' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chQsYSSDdI --notification-plan npTechnicalContactsEmail --label rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342 --criteria ' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }'", "delta": "0:00:01.292190", "end": "2015-12-11 00:48:06.935701", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "delta": "0:00:01.312386", "end": "2015-12-11 00:48:02.488470", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_msgs_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "module_name": "shell"}, "item": {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:01.176084", "stderr": "", "stdout": ""}, {"criteria": " :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric[\"rabbitmq_msgs_excl_notifications\"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, \"RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}\"); }", "name": "rabbitmq_msgs_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:05.643511"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 23, 'error_line': 1, 'error_position': 23, 'input': ' :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=23, pos=23\n :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }\n                       ^'}>
failed: [bc08a2dd-infra1_rabbit_mq_container-96c29a24] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:05.226314', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24"', u'rc': 1, 'item': {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:01.161495', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24"'}, u'start': u'2015-12-11 00:48:04.064819'}, {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chJs9elVwM --notification-plan npTechnicalContactsEmail --label rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24 --criteria ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }'", "delta": "0:00:00.938386", "end": "2015-12-11 00:48:07.708600", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "delta": "0:00:01.161495", "end": "2015-12-11 00:48:05.226314", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-96c29a24\"", "module_name": "shell"}, "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:04.064819", "stderr": "", "stdout": ""}, {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:06.770214"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 22, 'error_line': 1, 'error_position': 22, 'input': ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=22, pos=22\n:set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }\n                      ^'}>
ecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=23, pos=23\n :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }\n                       ^'}>
failed: [bc08a2dd-infra1_rabbit_mq_container-ff443063] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:04.636171', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063"', u'rc': 1, 'item': {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:01.224140', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063"'}, u'start': u'2015-12-11 00:48:03.412031'}, {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chfYGkCra9 --notification-plan npTechnicalContactsEmail --label rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063 --criteria ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }'", "delta": "0:00:00.952600", "end": "2015-12-11 00:48:07.900390", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "delta": "0:00:01.224140", "end": "2015-12-11 00:48:04.636171", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-ff443063\"", "module_name": "shell"}, "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:03.412031", "stderr": "", "stdout": ""}, {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:06.947790"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 22, 'error_line': 1, 'error_position': 22, 'input': ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=22, pos=22\n:set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }\n                      ^'}>
ecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=23, pos=23\n :set consecutiveCount={# maas_alarm_local_consecutive_count #} if (metric["rabbitmq_msgs_excl_notifications"] > {# rabbitmq_queued_messages_excluding_notifications_threshold #} ) { return new AlarmStatus(CRITICAL, "RabbitMQ sum of queued messages excluding notifications queues is reaching configured limit. Currently above {# rabbitmq_queued_messages_excluding_notifications_threshold #}"); }\n                       ^'}>
failed: [bc08a2dd-infra1_rabbit_mq_container-1ae54342] => (item=[{u'changed': True, u'end': u'2015-12-11 00:48:05.053890', u'stdout': u'', u'cmd': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342"', u'rc': 1, 'item': {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}, u'stderr': u'', u'delta': u'0:00:02.362091', 'invocation': {'module_name': 'shell', 'module_args': u'raxmon-alarms-list --entity-id enV52EnfGx | grep "label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342"'}, u'start': u'2015-12-11 00:48:02.691799'}, {'name': 'rabbitmq_qgrowth_excl_notifications', 'criteria': u':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }'}]) => {"changed": true, "cmd": "raxmon-alarms-create --entity-id enV52EnfGx --check-id chQsYSSDdI --notification-plan npTechnicalContactsEmail --label rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342 --criteria ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }'", "delta": "0:00:01.603925", "end": "2015-12-11 00:48:08.692220", "item": [{"changed": true, "cmd": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "delta": "0:00:02.362091", "end": "2015-12-11 00:48:05.053890", "invocation": {"module_args": "raxmon-alarms-list --entity-id enV52EnfGx | grep \"label=rabbitmq_qgrowth_excl_notifications--bc08a2dd-infra1_rabbit_mq_container-1ae54342\"", "module_name": "shell"}, "item": {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}, "rc": 1, "start": "2015-12-11 00:48:02.691799", "stderr": "", "stdout": ""}, {"criteria": ":set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric[\"rabbitmq_msgs_excl_notifications\"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, \"RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}\"); }", "name": "rabbitmq_qgrowth_excl_notifications"}], "rc": 1, "start": "2015-12-11 00:48:07.088295"}
stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/raxmon_cli/common.py", line 130, in run_action
    callback(instance, options, args, done)
  File "/usr/local/bin/raxmon-alarms-create", line 39, in callback
    why=options.why)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 485, in create_alarm
    data=data, coerce=self.get_alarm, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 301, in _create
    headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 152, in request
    raw=raw
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/openstack.py", line 577, in request
    return super(OpenStackBaseConnection, self).request(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 683, in request
    response = responseCls(**kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libcloud/common/base.py", line 116, in __init__
    raise Exception(self.parse_error())
  File "/usr/local/lib/python2.7/dist-packages/rackspace_monitoring/drivers/rackspace.py", line 107, in parse_error
    raise error
RackspaceMonitoringValidationError: <ValidationError type=alarmParseError, message="Failed to parse alarm", details={'error_token': 'endOfLine', 'error_column': 22, 'error_line': 1, 'error_position': 22, 'input': ':set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }', 'message': 'parse error: failed to match \'endOfLine\' at line=1, col=22, pos=22\n:set consecutiveCount={# maas_alarm_local_consecutive_count #} if (rate(metric["rabbitmq_msgs_excl_notifications"]) > {#rabbitmq_queue_growth_rate_threshold / maas_check_period#}) { return new AlarmStatus(CRITICAL, "RabbitMQ Queue growth rate is above configured threshold. Currently above {# rabbitmq_queue_growth_rate_threshold #}"); }\n                      ^'}>
sigmavirus24 commented 8 years ago

I'm willing to bet that the alarm criteria wasn't properly templated but it's too late right now for me to look into this.

@mancdaz @hughsaunders can you look at this tomorrow?

andymcc commented 8 years ago

@byronmccollum It looks like you're missing some new vars, can you ensure the following are set:

The problem is that in Juno we don't default vars (because the use of defaults/main.yml only really came into play in the kilo restructure). Unfortunately that means there is a manual process in Juno, that we haven't done a good job of documenting at this point.

rabbitmq_queued_messages_excluding_notifications_threshold: 100 rabbitmq_queue_growth_rate_threshold: 15

andymcc commented 8 years ago

If that doesn't resolve the problem let me know and I'll check it out!

byronmccollum commented 8 years ago

That was it. Thanks.

andymcc commented 8 years ago

@asettle @robb-romans can you think of a good way we can document this? The issue is essentially:

In juno, when we add new vars there is no way to automatically copy them, so during upgrades we need to copy over new vars (in etc/rpc_deploy/user_variables.yml) for some monitors (the above 2 in particular for this example) - otherwise the vars aren't specified. This is only an issue for Juno, and more specifically only for juno upgrades.

sigmavirus24 commented 8 years ago

more specifically only for juno upgrades.

To be clear, juno -> juno upgrades (e.g., 10.1.17 -> 10.1.18)

asettle commented 8 years ago

@andymcc - no problem. We can just add a note in the current upgrade guide and in the release notes. Clarifying that this is just for Juno and not relevant for the Kilo upgrades?

asettle commented 8 years ago

Change that, just the v10 release notes should be fine. See here: http://docs.rackspace.com/rpc/api/v10/bk-rpc-v10-releasenotes/content/rpc-upgrade.html

robb-romans commented 8 years ago

also: https://github.com/rpcdocs/rpcdocs/issues/449

git-harry commented 8 years ago

This repo is no longer actively maintained. If this issue is still valid please open a new issue at https://github.com/rcbops/rpc-openstack/issues