Closed larihuttunen closed 7 years ago
It seems Shadowserver has sent out a batch of broken CSV files with new-line chracters in unquoted fields.
2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Traceback (most recent call last): 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 313, in execute 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL return self.run() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 421, in run 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL return idiokit.main_loop(throw_stop_on_signal() | self._run()) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/bot.py", line 410, in _run 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield self.lobby.offer(self.bot_name, service) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 229, in offer 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield self.fork() | service.run() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 307, in run 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL state = yield self.errors | self.kill_sessions() | self.main(state) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL value = yield next() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 325, in kill_sessions 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield idiokit.consume() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL value = yield next() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 735, in map 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL value = yield next() 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/services.py", line 340, in _guarded 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL state = yield session 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 229, in poll 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield self.fetch_mails(self.filter) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 308, in fetch_mails 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield self.handle(parts) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/imapbot.py", line 328, in handle 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL skip_rest = yield handler(headers, fileobj) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 124, in handle_text_plain 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL result = yield self.parse_csv(headers, filename, fileobj) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 72, in parse_csv 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL self.normalize(subject, match.groupdict())) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 352, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.throw(*args)) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/shadowservermail.py", line 60, in normalize 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL yield idiokit.send(event) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/idiokit/idiokit.py", line 354, in _next 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL next = require_stream(self._gen.send(peel_args(args))) 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 352, in csv_to_events 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL for row in _CSVReader(fileobj, charset=charset, delimiter=delimiter): 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL File "/usr/lib/python2.6/site-packages/abusehelper/core/utils.py", line 339, in __iter__ 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL for row in reader: 2016-11-30 05:39:45Z shadowservermail[8456] CRITICAL Error: new-line character seen in unquoted field - do you need to open the file in universal-newline mode?
This was an issue on the feeder side.
It seems Shadowserver has sent out a batch of broken CSV files with new-line chracters in unquoted fields.