search

mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources
https://www.mlsecproject.org/
GNU General Public License v3.0
653 stars 179 forks source link

IndexError #53

Closed krmaxwell closed 10 years ago

krmaxwell commented 10 years ago

While working on repro for #49 got:

(venv)kmaxwell@newton:~/src/combine$ python thresher.py
Loading raw feed data from harvest.json
[...]
Parsing feed from http://www.autoshun.org/files/shunlist.csv
Traceback (most recent call last):
  File "thresher.py", line 189, in <module>
    thresh('harvest.json', 'crop.json')
  File "thresher.py", line 166, in thresh
    harvest += thresher_map[site](response[2], response[0], 'inbound')
  File "thresher.py", line 108, in process_autoshun
    date = line.split(',')[1].split()[0]
IndexError: list index out of range
jedisct1 commented 10 years ago

Saw this issue as well.

But testing for a specific error message might not be a good way to fix this.

Feeds (all of them) can fail for many different reasons, and return a variety of error messages, from Nginx errors to MySQL errors.

krmaxwell commented 10 years ago

@jedisct1: Indeed and I really need to refactor the whole "raw text parsing" thing soon. That's the proper fix. This is just a bandage to stop the bleeding.

alexcpsec commented 10 years ago

@technoskald I believe when the "plugin" infra (#23) is implemented we should take better care of this, no?

krmaxwell commented 10 years ago

Yeah, that too.