DatabaseModifiedError

[hassanakbar4]

component_MailArchive: User Interface resolution_wontfix type_defect | by rcross@amsl.com

---

Long running searches may result in a DatabaseModifiedError

Apparently this is a known issue in Xapian, If the index changes too much during the query this error is raised. It is mode likely to happen during long running queries and/or during periods of high index update volume. Lately, this has been triggered almost daily during export queries.

An old term example: Yahoo.com

DatabaseModifiedError: The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation

---

Issue migrated from trac:1958 at 2021-09-22 16:47:46 +0500

[hassanakbar4]

@hassanakbar4 commented

---

Note to self. There are other examples. Search email for above error.

[hassanakbar4]

@hassanakbar4 edited the issue description

[hassanakbar4]

@hassanakbar4 changed title from Search term causes DatabaseModifiedError to DatabaseModifiedError

[hassanakbar4]

@hassanakbar4 changed priority from critical to medium

[hassanakbar4]

@hassanakbar4 commented

---

Not sure if this ticket is actionable. It would be good to know if Elasticsearch has the same issue or not.

[hassanakbar4]

@hassanakbar4 commented

---

See page 18 in Getting Started Guide for more information https://media.readthedocs.org/pdf/getting-started-with-xapian/latest/getting-started-with-xapian.pdf

[hassanakbar4]

@hassanakbar4 commented

---

04/18 16:22 got errors not many messages coming in data/log/archive_mail a whole bunch of index updates in /var/log/celery/worker1.log

[hassanakbar4]

@hassanakbar4 commented

---

This error occurs here:

... File "/a/mailarch/current/haystack/backends/xapian_backend.py" in _get_enquire_mset

1264. return enquire.get_mset(start_offset, end_offset, checkatleast)

Exception Type: DatabaseModifiedError at /arch/search/ Exception Value: The revision being read has been discarded - you should call Xapian::Database::reopen() and retry the operation

Presumably because the index has changed between the time the enquire() is initialized and get_mset completes.

[hassanakbar4]

@hassanakbar4 commented

---

v1.10.14 includes two changes to address this issue until Elasticsearch is deployed:

• eliminate double index updates for incoming messages
• retry queries up to 5x in the event of a DatabaseModified error

[hassanakbar4]

@hassanakbar4 changed status from new to closed

[hassanakbar4]

@hassanakbar4 changed resolution from ` towontfix`

[hassanakbar4]

@hassanakbar4 commented

---

See here: https://getting-started-with-xapian.readthedocs.io/en/latest/concepts/indexing/databases.html#concurrent-access

Basically this is the way Xapian works. Two changes to the index during a query will result in this error. It's not designed to handle frequent updates. Elasticsearch does not have this limitation.