mysociety / popit

DEPRECATED - Development on PopIt has stopped and it is no longer being maintained
https://goo.gl/Vvej4Q
Other
76 stars 33 forks source link

reindexing large PopIt instances exhausts Elasticsearch's memory #782

Open mhl opened 9 years ago

mhl commented 9 years ago

To quote @struan:

[T]he re-indexing uses the elasticsearch's bulk re-index facility so it's throwing all the json for each collection at elasticsearch in one go. We did this as re-indexing each document individually caused everything to crash to a halt. It may be that we need to do split up the collections a bit as a first stage and then re-index those bits.

Related: