Non-atomic option for buildwatson

etianen / django-watson

Full-text multi-table search application for Django. Easy to install and use, with good performance.

BSD 3-Clause "New" or "Revised" License

1.2k stars 130 forks source link

Non-atomic option for buildwatson #229

Closed dustinrb closed 6 years ago

dustinrb commented 6 years ago

Although having buildwatson add SearchEntries entries atomically is preferable, adding batches of SearchEntries to the database can help the index build successfully complete for large indexes (>1 GB).

This PR adds two flags to buildwatson:

--non_atomic: Submits each batch in a non-atomic transaction. It will create and incomplete index if buildwatson fails but may help with make large indexing jobs more stable.
--batch_size=100: Changes the batch size buildwatson uses to add entries to the database.

amureki commented 6 years ago

@dustinrb @etianen wiki/docs update would be good for this one. :)

dustinrb commented 6 years ago

I'm happy to write some brief documentation. I can expand the "Manually updating the search indexes" subsection of the Registering Models section on the wiki. Let me know if you can think of a better place to put this information.

etianen commented 6 years ago

Maybe a new wiki page dedicated to management commands would be good?

On 11 December 2017 at 23:19, Dustin Broderick notifications@github.com wrote:

I'm happy to write some brief documentation. I can expand the "Manually updating the search indexes" subsection of the Registering Models section on the wiki. Let me know if you can think of a better place to put this information.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/etianen/django-watson/pull/229#issuecomment-350891074, or mute the thread https://github.com/notifications/unsubscribe-auth/AAJFCI9p76wSt5NqFFRvCEeEjhrSuzMgks5s_biUgaJpZM4Q4Yic .