search

csirtgadgets / csirtg-indicator-py-v1

python implementation of the indicator-protocol
https://github.com/csirtgadgets/indicator-protocol
Mozilla Public License 2.0
7 stars 13 forks source link

KeyError from csirtg-smrt indicator.py #61

Closed ghost closed 7 years ago

ghost commented 7 years ago

Today something seems to be strange within the phishtank feed causing a KeyError: Logs:

Feb 09 10:25:16 localhorstt csirtg-smrt[14590]: 2017-02-09 10:25:16,920 - INFO - csirtg_smrt.archiver[115] - Cached provider phishtank.com in memory, 0 objects
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]: Process Process-1:
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]: Traceback (most recent call last):
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:   File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:     self.run()
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:   File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:     self._target(*self._args, **self._kwargs)
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:   File "/etc/cif/cif-venv/local/lib/python2.7/site-packages/csirtg_smrt/smrt.py", line 252, in _run_smrt
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:     for i in s.process(r, f, limit=args.limit, data=data, filters=filters):
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:   File "/etc/cif/cif-venv/local/lib/python2.7/site-packages/csirtg_smrt/smrt.py", line 222, in process
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:     yield list(i.format_keys())[0]
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:   File "build/bdist.linux-x86_64/egg/csirtg_indicator/indicator.py", line 119, in format_keys
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]:     d[k] = d[k].format(**d)
Feb 09 10:27:20 localhorstt csirtg-smrt[14590]: KeyError: u'dong'
Feb 09 11:24:36 localhorstt csirtg-smrt[14590]: 2017-02-09 11:24:36,248 - INFO - csirtg_smrt.archiver[105] - Caching archived indicators for provider dataplane.org

Annoying: SMRT stops consuming other feed provider and wait until next run stoping at with the same error.

File: -rw-r--r-- 1 cif cif 3834649 Feb 9 12:00 online-valid.json.gz

Content containing 'dong':

zcat online-valid.json.gz | json_pp | grep dong
      "url" : "http://junghee.andongit.com/config/info/CMD/Websec/512e50af0ac5d63c0c1a455d6ba9c57b/?dispatch=V7yFsXwY2Vj6AgYbcaeLTjdmPFoWiPWdWuTosqnumGBWWz88Km&email=",
      "url" : "http://junghee.andongit.com/config/info/CMD/Websec/512e50af0ac5d63c0c1a455d6ba9c57b/?dispatch=V7yFsXwY2Vj6AgYbcaeLTjdmPFoWiPWdWuTosqnumGBWWz88Km&email=",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/thankyou.php",
      "url" : "http://cardoctormobile.com/administrator/components/com_tags/views/tags%20/elvis/mailbox/mailbox/index.php?email=abuse@batdongsantphcm.com",
      "url" : "http://refreshdharan.com/bg/excel2/index.php?userid={dong.keonkwonfinancialconsultd@yahoo.com}",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/index.php",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/index.php?",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/index.php?email=abuse@123india.com",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/index.php?email=abuse@ril.com",
      "url" : "http://mariomanadong.com/Global/mailbox/mailbox/index.php?email=abuse@ril.com",
      "url" : "http://www.adamdennis.info/wp-includes/mailbox/index.php?rand=13InboxLightaspxn.1774256418&fid.4.1252899642&fid=1&fav.1&rand.13InboxLight.aspxn.1774256418&fid.1252899642&fid.1&fav.1&email=abuse@dongbu.com&.rand=13vqcr8bp0gud&lc=1033&id=64855&mkt=en-us&cbcxt=mai&snsc=1",
      "url" : "http://www.adamdennis.info/wp-includes/mailbox/index.php?rand=13InboxLightaspxn.1774256418&fid.4.1252899642&fid=1&fav.1&rand.13InboxLight.aspxn.1774256418&fid.1252899642&fid.1&fav.1&email=abuse@dongbu.com&.rand=13vqcr8bp0gud&lc=1033&id=64855&mkt=en-us&cbcxt=mai&snsc=1",
      "url" : "http://dongsuh.net/master/index.html/?/mastercard.com.br/programa/de/pontos/cadastro.html",
      "url" : "http://dongsuh.net/visa/index.html/?/visa.com.br/programa/de/pontos/cadastro.html",
      "url" : "http://dongphuccaocap.net/images/android/private/index.html",
      "url" : "http://dongphuccaocap.net/images/android/private/",
      "url" : "http://www.dongphuccaocap.net/images/android/private/"
      "url" : "http://chefdongazz.com/ddocs/ddocs/ddocs/index.htm",
      "url" : "http://chefdongazz.com/ddocs/ddocs/ddocs/",
      "url" : "http://www.chefdongazz.com/ddocs/ddocs/ddocs/",

Might this line cause the error? "url" : "http://refreshdharan.com/bg/excel2/index.php?userid={dong.keonkwonfinancialconsultd@yahoo.com}",

wesyoung commented 7 years ago

we added some functionality where you can do things like {var} and suck that {var} in from another key somewhere in the config.. didn't think about literal {}'s in the url :)

should be fixed.