Problem to use custom Rule

[leandro-ss]

Hi, I having problems to use your plugin,

I want use to take avg of time_take of httpd, but starts with this problem bellow...

Did you kwon whats happening??

INFO:elastalert:{u'time_taken_avg': {u'value': 576.4656878391319}, u'key': u'localhost.localdomain', u'doc_count': 6278} '''ERROR:root:Traceback (most recent call last): File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 1078, in run_all_rules num_matches = self.run_rule(rule, endtime, self.starttime) File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 781, in run_rule if not self.run_query(rule, rule['starttime'], tmp_endtime): File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 598, in run_query rule_inst.add_aggregation_data(data) File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 925, in add_aggregation_data self.unwrap_term_buckets(timestamp, payload_data['bucket_aggs']['buckets']) File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 939, in unwrap_term_buckets self.check_matches(timestamp, term_data['key'], term_data) File "custom/custom_rule2.py", line 46, in check_matches self.handle_event(aggregation_data, metric_val) File "custom/custom_rule2.py", line 62, in handle_event self.cur_windows[qk].append((event, value)) File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 319, in append while self.duration() >= self.timeframe: File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 329, in duration return self.get_ts(self.data[-1]) - self.get_ts(self.data[0]) TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'

ERROR:root:Uncaught exception running rule custom_spike_aggs: unsupported operand type(s) for -: 'NoneType' and'NoneType' INFO:elastalert:Skipping writing to ES: {'message': "Uncaught exception running rule custom_spike_aggs: unsupported operand type(s) for -: 'NoneType' and 'NoneType'", 'traceback': ['Traceback (most recent call last):', ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 1078, in run_all_rules', ' num_matches = self.run_rule(rule, endtime, self.starttime)', ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 781, in run_rule', " if not self.run_query(rule, rule['starttime'], tmp_endtime):", ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/elastalert.py", line 598, in run_query', ' rule_inst.add_aggregation_data(data)', ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 925, in add_aggregation_data', " self.unwrap_term_buckets(timestamp, payload_data['bucket_aggs']['buckets'])", ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 939, in unwrap_term_buckets', " self.check_matches(timestamp, term_data['key'], term_data)", ' File "custom/custom_rule2.py", line 46, in check_matches', ' self.handle_event(aggregation_data, metric_val)', ' File "custom/custom_rule2.py", line 62, in handle_event', ' self.cur_windows[qk].append((event, value))', ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 319, in append', ' while self.duration() >= self.timeframe:', ' File "/home/inmetrics/gitHub/project_tutorial_elastalert/venv/local/lib/python2.7/site-packages/elastalert-0.1.21-py2.7.egg/elastalert/ruletypes.py", line 329, in duration', ' return self.get_ts(self.data[-1]) - self.get_ts(self.data[0])', "TypeError: unsupported operand type(s) for -: 'NoneType' and 'NoneType'"], 'data': {'rule': 'custom_spike_aggs'}} INFO:elastalert:Rule custom_spike_aggs disabled '''

[leandro-ss]

I did some changes and I got to run, but didn't generate alert.

Did you really use that code??

[phillbaker]

Hi @leandro-ss, I apologize this commit was definitely work in progress, I think it was extracted at a mid point of our work and I never updated things. If you'd like to PR your changes, happy to take a look and compare notes.

[leandro-ss]

Ok @phillbaker ,

But did you do some change after that in branch??

The idea of extends direct of BaseAggregationRule is valid? Why not MetricAggregationRule?

Tks

[leandro-ss]

How did you resolved your problem with alerts?? Did you adopted other tool??

[dylanjf]

https://github.com/dylanjf/elastalert

I forked a working solution to this, which creates a new alert type spike_aggregation

• metric_agg_type lets you set the aggregation (min, max, avg, sum, cardinality, value_count)
• metric_agg_script lets you define a custom on-the-fly metric
• min_doc_count lets you only consider groups defined in query_key with at least X records over the past time period

We're currently using this against our advertising logs to track spikes in revenue from partners and percentage of ads rendered on our pages.

[phillbaker]

Nice, looks like you were able to PR it and it's merged! https://github.com/Yelp/elastalert/pull/2052

I'll archive this repo. Thanks for the update @dylanjf