modoboa / modoboa-stats

Graphical statistics for Modoboa
MIT License
9 stars 13 forks source link

FIX: UnicodeDecodeError during logfile parsing #57

Open tsabi opened 4 years ago

tsabi commented 4 years ago

If you use postfix internationally, you will hit this error. I believe this is the right fix, because those log files are NOT utf-8 encoded, so you should not read it as utf-8, but with the real encoding of that file.

Fixes: https://github.com/modoboa/modoboa/issues/1788, https://github.com/modoboa/modoboa-stats/issues/55, https://github.com/modoboa/modoboa-stats/issues/38,

root@h7-mailsrv:/etc/cron.d# /srv/modoboa/env/bin/python /srv/modoboa/instance/manage.py logparser
Traceback (most recent call last):
  File "/srv/modoboa/instance/manage.py", line 22, in <module>
    execute_from_command_line(sys.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 587, in handle
    p.process()
  File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_stats/management/commands/logparser.py", line 552, in process
    for line in self.f.readlines():
  File "/srv/modoboa/env/lib/python2.7/codecs.py", line 314, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc1 in position 949: invalid start byte
mundschenk-at commented 3 years ago

I got a similar error tonight: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 2515: invalid continuation byte. The rought cause in the logfile appears to be an issue with an invalid domain name.

postfix/trivial-rewrite[49231]: warning: midna_domain_to_ascii_create: Problem translating domain [...] to ASCII form: UIDNA_ERROR_DISALLOWED

I don't think opening logfiles as latin1 is the correct fix, but the logparser should handle unicode exceptions.