abusesa / abusehelper

A framework for receiving and redistributing abuse feeds
MIT License
118 stars 18 forks source link

CSVReader Error Handling #83

Closed larihuttunen closed 7 years ago

larihuttunen commented 7 years ago

It seems Shadowserver has sent out a batch of broken CSV files with new-line chracters in unquoted fields.

2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Traceback (most recent call last):
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 313, in execute
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     return self.run()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 421, in run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     return idiokit.main_loop(throw_stop_on_signal() | self._run())
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 410, in _run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.lobby.offer(self.bot_name, service)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 229, in offer
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.fork() | service.run()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 307, in run
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     state = yield self.errors | self.kill_sessions() | self.main(state)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 325, in kill_sessions
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield idiokit.consume()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     value = yield next()
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 340, in _guarded
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     state = yield session
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 229, in poll
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.fetch_mails(self.filter)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 308, in fetch_mails
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield self.handle(parts)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 328, in handle
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     skip_rest = yield handler(headers, fileobj)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 124, in handle_text_plain
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     result = yield self.parse_csv(headers, filename, fileobj)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 72, in parse_csv
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     self.normalize(subject, match.groupdict()))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.throw(*args))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 60, in normalize
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     yield idiokit.send(event)
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 354, in _next
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     next = require_stream(self._gen.send(peel_args(args)))
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 352, in csv_to_events
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     for row in _CSVReader(fileobj, charset=charset, delimiter=delimiter):
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL   File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 339, in __iter__
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL     for row in reader:
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
larihuttunen commented 7 years ago

This was an issue on the feeder side.