squelcher crashes

[aaronkaplan]

2015-11-03 14:38:58,240 - squelcher-expert - DEBUG - Receive message u'{"classification.taxonomy": "Malicious Code", "feed.url": "https://example.com/today", "time.observation": "2015-10-08T01:38:08+00:00", "raw": "XXXXXX=", "classification.identifier": "dga", "__type": "Event", "classification.type": "c&c", "source.fqdn": "tsvridclsswsow.me", "feed.name": "Fraunhofer DGA"}'...
2015-11-03 14:38:58,240 - squelcher-expert - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/lib/bot.py", line 97, in start
    self.process()
  File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/bots/experts/squelcher/expert.py", line 65, in process
    in_net = (ip_address(event['source.ip']) in
KeyError: u'source.ip'

I think the squelcher assumes that source.ip is always defined. But that really actually depends on the processing chain. In the case of domains arriving, we do not necessarily have an IP address. What to do? Can we define an "ignore" behaviour for the squelcher?

Like something like report in the log file, but continue?

[sebix]

If source.ip or source.asn does not exist, notify should be None.

[aaronkaplan]

even when settings error_procudure: "pass" the bot will exit.

2015-11-03 14:49:01,749 - squelcher-expert - DEBUG - Receiving Message.
2015-11-03 14:49:01,749 - squelcher-expert - DEBUG - Receive message u'{"classification.taxonomy": "Malicious Code", "feed.url": "https://example.com/today", "time.observation": "2015-10-08T01:38:08+00:00", "raw": "bWtiZWNtbGp0bmJmdmcuY2M=", "classification.identifier": "dga", "__type": "Event", "classification.type": "c&c", "source.fqdn": "mkbecmljtnbfvg.cc", "feed.name": "Fraunhofer DGA"}'...
2015-11-03 14:49:01,750 - squelcher-expert - ERROR - Bot has found a problem.
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/lib/bot.py", line 97, in start
    self.process()
  File "/usr/local/lib/python2.7/dist-packages/intelmq-1.0.0-py2.7.egg/intelmq/bots/experts/squelcher/expert.py", line 65, in process
    in_net = (ip_address(event['source.ip']) in
KeyError: u'source.ip'
2015-11-03 14:49:01,750 - squelcher-expert - INFO - Last Correct Message(event): {u'classification.taxonomy': u'Malicious Code', u'feed.url': u'https://example.com/today', u'time.observation': u'2015-10-08T01:38:08+00:00', u'raw': u'b3FtcnFzcWhpam53b3AubWU=', u'classification.identifier': u'dga', u'classification.type': u'c&c', u'source.fqdn': u'oqmrqsqhijnwop.me', u'feed.name': u'Fraunhofer DGA'}.
2015-11-03 14:49:01,750 - squelcher-expert - INFO - Current Message(event): {u'classification.taxonomy': u'Malicious Code', u'feed.url': u'https://example.com/today', u'time.observation': u'2015-10-08T01:38:08+00:00', u'raw': u'bWtiZWNtbGp0bmJmdmcuY2M=', u'classification.identifier': u'dga', u'classification.type': u'c&c', u'source.fqdn': u'mkbecmljtnbfvg.cc', u'feed.name': u'Fraunhofer DGA'}.
2015-11-03 14:49:01,753 - squelcher-expert - INFO - Disconnecting from source pipeline.
2015-11-03 14:49:01,753 - squelcher-expert - INFO - Disconnecting from destination pipeline.
2015-11-03 14:49:01,753 - squelcher-expert - INFO - Bot stopped.
2015-11-03 14:49:01,753 - squelcher-expert - ERROR - Exiting.

[sebix]

This is the old error behavior, the last changes are not merged yet from certtools/intelmq#376

[sebix]

Should be solved with the merge it just did and the fix for #11.