Example fails: elastalert.util.EAException: Invalid Rule: Metricbeat CPU Spike Rule

[jonathan-kosgei]

I'm testing

name: Metricbeat CPU Spike Rule
type: metric_aggregation

index: metricbeat-*

buffer_time:
  hours: 1

metric_agg_key: system.cpu.user.pct
metric_agg_type: avg
query_key: beat.hostname
doc_type: metricsets

bucket_interval:
  minutes: 5

sync_bucket_interval: true

min_threshold: 0.1
max_threshold: 0.8

filter:
- term:
    metricset.name: cpu

alert:
- "email"

email:
- "alert@email.com"

But I get

Traceback (most recent call last):
  File "/usr/local/bin/elastalert-test-rule", line 11, in <module>
    load_entry_point('elastalert', 'console_scripts', 'elastalert-test-rule')()
  File "/opt/elastalert/elastalert/test_rule.py", line 328, in main
    test_instance.run_rule_test()
  File "/opt/elastalert/elastalert/test_rule.py", line 309, in run_rule_test
    conf = self.load_conf(rule_yaml, args)
  File "/opt/elastalert/elastalert/test_rule.py", line 288, in load_conf
    load_options(rules, conf)
  File "/opt/elastalert/elastalert/config.py", line 115, in load_options
    raise EAException("Invalid Rule: %s\n%s" % (rule.get('name'), e))
elastalert.util.EAException: Invalid Rule: Metricbeat CPU Spike Rule
{'index': 'metricbeat-*', 'name': 'Metricbeat CPU Spike Rule', 'sync_bucket_interval': True, 'query_key': 'beat.hostname', 'min_threshold': 0.1, 'max_threshold': 0.8, 'alert': ['email'], 'filter': [{'term': {'metricset.name': 'cpu'}}], 'bucket_interval': {'minutes': 5}, 'doc_type': 'metricsets', 'type': 'metric_aggregation', 'metric_agg_key': 'system.cpu.user.pct', 'buffer_time': {'hours': 1}, 'email': ['jonathan@saharacluster.com'], 'metric_agg_type': 'avg'} is not valid under any of the given schemas

Failed validating 'oneOf' in schema:
    {'$schema': 'http://json-schema.org/draft-04/schema#',
     'definitions': {'arrayOfStrings': {'items': {'type': 'string'},
                                        'type': ['string', 'array']},
                     'arrayOfStringsOrOtherArrays': {'items': {'type': ['string',
                                                                        'array']},
                                                     'type': ['string',
                                                              'array']},
                     'filter': {},
                     'timeFrame': {'additionalProperties': False,
                                   'properties': {'days': {'type': 'number'},
                                                  'hours': {'type': 'number'},
                                                  'milliseconds': {'type': 'number'},
                                                  'minutes': {'type': 'number'},
                                                  'schedule': {'type': 'string'},
                                                  'seconds': {'type': 'number'},
                                                  'weeks': {'type': 'number'}},
                                   'type': 'object'}},
     'oneOf': [{'properties': {'type': {'enum': ['any']}},
                'title': 'Any'},
               {'properties': {'blacklist': {'items': {'type': 'string'},
                                             'type': 'array'},
                               'compare_key': {'type': 'string'},
                               'type': {'enum': ['blacklist']}},
                'required': ['blacklist', 'compare_key'],
                'title': 'Blacklist'},
               {'properties': {'compare_key': {'type': 'string'},
                               'ignore_null': {'type': 'boolean'},
                               'type': {'enum': ['whitelist']},
                               'whitelist': {'items': {'type': 'string'},
                                             'type': 'array'}},
                'required': ['whitelist', 'compare_key', 'ignore_null'],
                'title': 'Whitelist'},
               {'properties': {'compare_key': {'type': 'string'},
                               'ignore_null': {'type': 'boolean'},
                               'timeframe': {'additionalProperties': False,
                                             'properties': {'days': {'type': 'number'},
                                                            'hours': {'type': 'number'},
                                                            'milliseconds': {'type': 'number'},
                                                            'minutes': {'type': 'number'},
                                                            'schedule': {'type': 'string'},
                                                            'seconds': {'type': 'number'},
                                                            'weeks': {'type': 'number'}},
                                             'type': 'object'},
                               'type': {'enum': ['change']}},
                'required': ['query_key', 'compare_key', 'ignore_null'],
                'title': 'Change'},
               {'properties': {'attach_related': {'type': 'boolean'},
                               'doc_type': {'type': 'string'},
                               'num_events': {'type': 'integer'},
                               'terms_size': {'type': 'integer'},
                               'timeframe': {'additionalProperties': False,
                                             'properties': {'days': {'type': 'number'},
                                                            'hours': {'type': 'number'},
                                                            'milliseconds': {'type': 'number'},
                                                            'minutes': {'type': 'number'},
                                                            'schedule': {'type': 'string'},
                                                            'seconds': {'type': 'number'},
                                                            'weeks': {'type': 'number'}},
                                             'type': 'object'},
                               'type': {'enum': ['frequency']},
                               'use_count_query': {'type': 'boolean'},
                               'use_terms_query': {'type': 'boolean'}},
                'required': ['num_events', 'timeframe'],
                'title': 'Frequency'},
               {'properties': {'alert_on_new_data': {'type': 'boolean'},
                               'doc_type': {'type': 'string'},
                               'spike_height': {'type': 'number'},
                               'spike_type': {'enum': ['up',
                                                       'down',
                                                       'both']},
                               'terms_size': {'type': 'integer'},
                               'threshold_cur': {'type': 'integer'},
                               'threshold_ref': {'type': 'integer'},
                               'timeframe': {'additionalProperties': False,
                                             'properties': {'days': {'type': 'number'},
                                                            'hours': {'type': 'number'},
                                                            'milliseconds': {'type': 'number'},
                                                            'minutes': {'type': 'number'},
                                                            'schedule': {'type': 'string'},
                                                            'seconds': {'type': 'number'},
                                                            'weeks': {'type': 'number'}},
                                             'type': 'object'},
                               'type': {'enum': ['spike']},
                               'use_count_query': {'type': 'boolean'},
                               'use_terms_query': {'type': 'boolean'}},
                'required': ['spike_height', 'spike_type', 'timeframe'],
                'title': 'Spike'},
               {'properties': {'doc_type': {'type': 'string'},
                               'threshold': {'type': 'integer'},
                               'timeframe': {'additionalProperties': False,
                                             'properties': {'days': {'type': 'number'},
                                                            'hours': {'type': 'number'},
                                                            'milliseconds': {'type': 'number'},
                                                            'minutes': {'type': 'number'},
                                                            'schedule': {'type': 'string'},
                                                            'seconds': {'type': 'number'},
                                                            'weeks': {'type': 'number'}},
                                             'type': 'object'},
                               'type': {'enum': ['flatline']},
                               'use_count_query': {'type': 'boolean'}},
                'required': ['threshold', 'timeframe'],
                'title': 'Flatline'},
               {'properties': {'alert_on_missing_field': {'type': 'boolean'},
                               'fields': {'items': {'type': ['string',
                                                             'array']},
                                          'type': ['string', 'array']},
                               'terms_size': {'type': 'integer'},
                               'terms_window_size': {'additionalProperties': False,
                                                     'properties': {'days': {'type': 'number'},
                                                                    'hours': {'type': 'number'},
                                                                    'milliseconds': {'type': 'number'},
                                                                    'minutes': {'type': 'number'},
                                                                    'schedule': {'type': 'string'},
                                                                    'seconds': {'type': 'number'},
                                                                    'weeks': {'type': 'number'}},
                                                     'type': 'object'},
                               'type': {'enum': ['new_term']},
                               'use_terms_query': {'type': 'boolean'}},
                'required': ['fields'],
                'title': 'New Term'},
               {'properties': {'cardinality_field': {'type': 'string'},
                               'max_cardinality': {'type': 'integer'},
                               'min_cardinality': {'type': 'integer'},
                               'timeframe': {'additionalProperties': False,
                                             'properties': {'days': {'type': 'number'},
                                                            'hours': {'type': 'number'},
                                                            'milliseconds': {'type': 'number'},
                                                            'minutes': {'type': 'number'},
                                                            'schedule': {'type': 'string'},
                                                            'seconds': {'type': 'number'},
                                                            'weeks': {'type': 'number'}},
                                             'type': 'object'},
                               'type': {'enum': ['cardinality']}},
                'required': ['cardinality_field', 'timeframe'],
                'title': 'Cardinality'},
               {'properties': {'type': {'pattern': '[.]'}},
                'title': 'Custom Rule from Module'}],
     'properties': {'aggregation': {'additionalProperties': False,
                                    'properties': {'days': {'type': 'number'},
                                                   'hours': {'type': 'number'},
                                                   'milliseconds': {'type': 'number'},
                                                   'minutes': {'type': 'number'},
                                                   'schedule': {'type': 'string'},
                                                   'seconds': {'type': 'number'},
                                                   'weeks': {'type': 'number'}},
                                    'type': 'object'},
                    'alert_text': {'type': 'string'},
                    'alert_text_args': {'items': {'type': 'string'},
                                        'type': 'array'},
                    'alert_text_kw': {'type': 'object'},
                    'alert_text_type': {'enum': ['alert_text_only',
                                                 'exclude_fields']},
                    'buffer_time': {'additionalProperties': False,
                                    'properties': {'days': {'type': 'number'},
                                                   'hours': {'type': 'number'},
                                                   'milliseconds': {'type': 'number'},
                                                   'minutes': {'type': 'number'},
                                                   'schedule': {'type': 'string'},
                                                   'seconds': {'type': 'number'},
                                                   'weeks': {'type': 'number'}},
                                    'type': 'object'},
                    'command': {'items': {'type': 'string'},
                                'type': ['string', 'array']},
                    'email': {'items': {'type': 'string'},
                              'type': ['string', 'array']},
                    'email_reply_to': {'type': 'string'},
                    'es_host': {'type': 'string'},
                    'es_password': {'type': 'string'},
                    'es_port': {'type': 'integer'},
                    'es_username': {'type': 'string'},
                    'exponential_realert': {'additionalProperties': False,
                                            'properties': {'days': {'type': 'number'},
                                                           'hours': {'type': 'number'},
                                                           'milliseconds': {'type': 'number'},
                                                           'minutes': {'type': 'number'},
                                                           'schedule': {'type': 'string'},
                                                           'seconds': {'type': 'number'},
                                                           'weeks': {'type': 'number'}},
                                            'type': 'object'},
                    'fail_on_non_zero_exit': {'type': 'boolean'},
                    'field': {},
                    'filter': {'additionalProperties': False,
                               'items': {},
                               'properties': {'download_dashboard': {'type': 'string'}},
                               'type': ['array', 'object']},
                    'from_addr': {'type': 'string'},
                    'generate_kibana_link': {'type': 'boolean'},
                    'gitter_msg_level': {'enum': ['info', 'error']},
                    'gitter_proxy': {'type': 'string'},
                    'gitter_webhook_url': {'type': 'string'},
                    'hipchat_auth_token': {'type': 'string'},
                    'hipchat_domain': {'type': 'string'},
                    'hipchat_from': {'type': 'string'},
                    'hipchat_ignore_ssl_errors': {'type': 'boolean'},
                    'hipchat_notify': {'type': 'boolean'},
                    'hipchat_room_id': {'type': 'string'},
                    'include': {'items': {'type': 'string'},
                                'type': 'array'},
                    'index': {'type': 'string'},
                    'jira_account_file': {'type': 'string'},
                    'jira_assignee': {'type': 'string'},
                    'jira_bump_in_statuses': {'items': {'type': 'string'},
                                              'type': ['string',
                                                       'array']},
                    'jira_bump_not_in_statuses': {'items': {'type': 'string'},
                                                  'type': ['string',
                                                           'array']},
                    'jira_bump_tickets': {'type': 'boolean'},
                    'jira_component': {'items': {'type': 'string'},
                                       'type': ['string', 'array']},
                    'jira_components': {'items': {'type': 'string'},
                                        'type': ['string', 'array']},
                    'jira_issuetype': {'type': 'string'},
                    'jira_label': {'items': {'type': 'string'},
                                   'type': ['string', 'array']},
                    'jira_labels': {'items': {'type': 'string'},
                                    'type': ['string', 'array']},
                    'jira_max_age': {'type': 'number'},
                    'jira_project': {'type': 'string'},
                    'jira_server': {'type': 'string'},
                    'jira_watchers': {'items': {'type': 'string'},
                                      'type': ['string', 'array']},
                    'kibana_dashboard': {'type': 'string'},
                    'match_enhancements': {'items': {'type': 'string'},
                                           'type': 'array'},
                    'max_query_size': {'type': 'integer'},
                    'name': {'type': 'string'},
                    'notify_email': {'items': {'type': 'string'},
                                     'type': ['string', 'array']},
                    'owner': {'type': 'string'},
                    'pagerduty_client_name': {'type': 'string'},
                    'pagerduty_service_key': {'type': 'string'},
                    'pipe_match_json': {'type': 'boolean'},
                    'priority': {'type': 'integer'},
                    'query_delay': {'additionalProperties': False,
                                    'properties': {'days': {'type': 'number'},
                                                   'hours': {'type': 'number'},
                                                   'milliseconds': {'type': 'number'},
                                                   'minutes': {'type': 'number'},
                                                   'schedule': {'type': 'string'},
                                                   'seconds': {'type': 'number'},
                                                   'weeks': {'type': 'number'}},
                                    'type': 'object'},
                    'query_key': {'items': {'type': 'string'},
                                  'type': ['string', 'array']},
                    'raw_count_keys': {'type': 'boolean'},
                    'realert': {'additionalProperties': False,
                                'properties': {'days': {'type': 'number'},
                                               'hours': {'type': 'number'},
                                               'milliseconds': {'type': 'number'},
                                               'minutes': {'type': 'number'},
                                               'schedule': {'type': 'string'},
                                               'seconds': {'type': 'number'},
                                               'weeks': {'type': 'number'}},
                                'type': 'object'},
                    'replace_dots_in_field_names': {'type': 'boolean'},
                    'slack_emoji_override': {'type': 'string'},
                    'slack_icon_url_override': {'type': 'string'},
                    'slack_msg_color': {'enum': ['good',
                                                 'warning',
                                                 'danger']},
                    'slack_parse_override': {'enum': ['none', 'full']},
                    'slack_text_string': {'type': 'string'},
                    'slack_username_override': {'type': 'string'},
                    'slack_webhook_url': {'items': {'type': 'string'},
                                          'type': ['string', 'array']},
                    'smtp_host': {'type': 'string'},
                    'telegram_api_url': {'type': 'string'},
                    'telegram_bot_token': {'type': 'string'},
                    'telegram_room_id': {'type': 'string'},
                    'timestamp_field': {'type': 'string'},
                    'top_count_keys': {'items': {'type': 'string'},
                                       'type': 'array'},
                    'top_count_number': {'type': 'integer'},
                    'twilio_accout_sid': {'type': 'string'},
                    'twilio_auth_token': {'type': 'string'},
                    'twilio_from_number': {'type': 'string'},
                    'twilio_to_number': {'type': 'string'},
                    'use_kibana_dashboard': {'type': 'string'},
                    'use_local_time': {'type': 'boolean'},
                    'use_ssl': {'type': 'boolean'},
                    'use_strftime_index': {'type': 'boolean'},
                    'verify_certs': {'type': 'boolean'},
                    'victorops_api_key': {'type': 'string'},
                    'victorops_entity_display_name': {'type': 'string'},
                    'victorops_message_type': {'enum': ['INFO',
                                                        'WARNING',
                                                        'ACKNOWLEDGEMENT',
                                                        'CRITICAL',
                                                        'RECOVERY']},
                    'victorops_routing_key': {'type': 'string'}},
     'required': ['name', 'type', 'index', 'alert'],
     'type': 'object'}

On instance:
    {'alert': ['email'],
     'bucket_interval': {'minutes': 5},
     'buffer_time': {'hours': 1},
     'doc_type': 'metricsets',
     'email': ['jonathan@saharacluster.com'],
     'filter': [{'term': {'metricset.name': 'cpu'}}],
     'index': 'metricbeat-*',
     'max_threshold': 0.8,
     'metric_agg_key': 'system.cpu.user.pct',
     'metric_agg_type': 'avg',
     'min_threshold': 0.1,
     'name': 'Metricbeat CPU Spike Rule',
     'query_key': 'beat.hostname',
     'sync_bucket_interval': True,
     'type': 'metric_aggregation'}

I'm working from the example at https://github.com/Yelp/elastalert/blob/master/example_rules/example_single_metric_agg.yaml

[Qmando]

You seem to be using too old of a version.

[Rambabuyendru520]

Hi Jonathan-kosgei, How did you solve this issue, i am also facing same issue,please help me...

[Qmando]

@Rambabuyendru520 Can you post your rule? If you got a jsonschema error you are either missing something or have some invalid setting

[Rambabuyendru520]

Hi @Qmando, thank you for responding, I have to resolved my issue..

[yatam-manasa]

Hi @Rambabuyendru520 .How did you solve this issue?