We could implement a fallback strategy in such cases and try to commit each single document of that batch separately to decrease the amount of affected documents.
Also logging should be improved a bit, currently a failing batch will log:
2020-03-27 12:11:36 ERROR ftw.solr.connection Update command failed. Server responded with code 400, Invalid Date in Date Math String:'2015-04-07T14:43:60.000Z'.
2020-03-27 12:11:45 Intermediate commit (63 items processed, last batch in 122.644s)
Especially
2020-03-27 12:11:45 Intermediate commit (63 items processed, last batch in 122.644s)
is misleading (at least for me) as it form the message it is non-obvious that the whole batch actually failed to commit.
njohner
commented
4 years ago
This may be by design, but is has caused issues for Solr activation and Solr maintenance.
If a commit in
syncorindexat https://github.com/4teamwork/ftw.solr/blob/8354f7658a9e301854d671e3d478923509fa55fe/ftw/solr/browser/maintenance.py#L251 or https://github.com/4teamwork/ftw.solr/blob/8354f7658a9e301854d671e3d478923509fa55fe/ftw/solr/browser/maintenance.py#L100 fails this will lead to the whole batch of documents being discarded.We could implement a fallback strategy in such cases and try to commit each single document of that batch separately to decrease the amount of affected documents.
Also logging should be improved a bit, currently a failing batch will log:
Especially
is misleading (at least for me) as it form the message it is non-obvious that the whole batch actually failed to commit.