report_id for yahoo is truncated and rounded

[tschuettler]

The actual report_id 1536287116.249533 from the xml gets truncated and rounded to 1536287116.25.

Since we got 9 reports from yahoo that all have the about the same report_id (=timestamp) and were received at about the same time there was a unique contraint violation when the truncated report_id did not equal the rounded report_id that is stored from a previous report.

Since this exception is not handled, the full traceback will be send to the reporter.

Traceback for: Report Domain: email.oxfam.de Submitter: compuserve.com Report-ID: <1536287116.249533>:

Importing report 1536287116.25 received from Yahoo! Inc.
Traceback (most recent call last): File "/srv/modoboa/instance/manage.py", line 22, in <module> execute_from_command_line(sys.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line utility.execute()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/__init__.py", line 356, in execute self.fetch_command(subcommand).run_from_argv(self.argv)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 283, in run_from_argv self.execute(*args, **cmd_options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/core/management/base.py", line 330, in execute output = self.handle(*args, **options)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_dmarc/management/commands/import_aggregated_report.py", line 36, in handle lib.import_report_from_stdin()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_dmarc/lib.py", line 171, in import_report_from_stdin import_report_from_email(content)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_dmarc/lib.py", line 149, in import_report_from_email import_archive(fpo, content_type=part.get_content_type())
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_dmarc/lib.py", line 130, in import_archive import_report(zfile.read())
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/utils/decorators.py", line 185, in inner return func(*args, **kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/modoboa_dmarc/lib.py", line 112, in import_report report.save()
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save force_update=force_update, update_fields=update_fields)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 838, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 924, in _save_table result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 963, in _do_insert using=using, raw=raw)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/query.py", line 1076, in _insert return query.get_compiler(using=using).execute_sql(return_id)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1112, in execute_sql cursor.execute(sql, params)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback)
File "/srv/modoboa/env/local/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: duplicate key value violates unique constraint "modoboa_dmarc_report_reporter_id_431d125f_uniq" DETAIL: Key (reporter_id, report_id)=(3, 1536287116.25) already exists.

The reports where the report_id is only truncated and rounded down are silently ignored when there is a matching report found, that should not actually be matched: (Importing report 1536287116.25 received from Yahoo! Inc. Report already imported.)

[tschuettler]

I made my first steps with python and tried to debug this.

I was able to reproduce this behaviour with the test report https://github.com/modoboa/modoboa-dmarc/blob/master/modoboa_dmarc/tests/reports/Report_Domain_ngyn.org_Submitter_yahoo.com_Report-ID_1435111091.916236.eml:

The report_id is stored with less precission as a float.

Python3 seems to have a higher precission for floats as it is working fine there. I'm not sure where to alter that default behaviour. But since python3 does not seem to be affected i will try to upgrade the virtualenv.

I could boil the issue down to 2 minimal virtualenvs:

Python 2.7.12 (default, Dec  4 2017, 14:50:18)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(float('1435111091.916236'))
1435111091.92

Python 3.5.2 (default, Nov 12 2018, 13:43:14)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print(float('1435111091.916236'))
1435111091.916236

[tonioo]

@tschuettler Sorry for the late reply. Actually, modoboa does not try to interpret the report ID (you can see the type of the model field is Charfield). Unfortunately, I think it is an lxml issue which tries to guess the type of attribute. I don't know if there is an easy way to fix this on Python2...

[tschuettler]

The issue is now obsolete since Python2 is not supported anymore.