search

4teamwork / ftw.solr

Solr integration for Plone
5 stars 5 forks source link

Rounding error for DateTime with milliseconds >= 999501 #157

Closed deiferni closed 4 years ago

deiferni commented 4 years ago

In https://github.com/4teamwork/ftw.solr/blob/16f13492f9c991eb3cb07ab8a5ce32452d5ecbc8/ftw/solr/converters.py#L7-L11 we convert DateTime to an ISO 8601 date format string. The code currently contains rounding errors so that any DateTime with milliseconds >= 999501 will be converted to an ISO 8601 date format string with the string 60 in the [ss] part of the date format e.g.:

to_iso8601(DateTime('2015/04/07 16:43:59.999501'))

Will result in:

u'2015-04-07T14:43:60.000Z'

Solr does not enjoy this at all and indexing will result in an error:

2020-03-26 15:13:28 ERROR ftw.solr.connection Update command failed. Server responded with code 400, Invalid Date in Date Math String:'2015-04-07T14:43:60.000Z'.

This has the consequence that statistically 5‰ of the documents in solr are not indexed, or not updated correctly when ftw.solr is used. If there are more than one DateTime based date fields in solr multiply the value by the number of affected fields.

deiferni commented 4 years ago

Fixed by https://github.com/4teamwork/ftw.solr/pull/158