These changes appear to make the reindexing work without timing out. I think the main issue was that all four models we're being reindexed at once, so they were all treading on each others toes. By using mapSeries they run one model at a time, which seems to reduce load (though memory usage is still quite high during the reindex?).
chrism@ajax:/data/vhost/popit.staging.mysociety.org/popit$ time bin/replace-database ~mark/popolo-test-mongo- popit_staging_mark-test
...
Reindexing the database: popit_staging_mark-test
Sending elasticsearch bulk command for Person
Completed elasticsearch bulk command for Person
Sending elasticsearch bulk command for Organization
Completed elasticsearch bulk command for Organization
Sending elasticsearch bulk command for Membership
Completed elasticsearch bulk command for Membership
Re-indexed 28657 docs from popit_staging_mark-test
real 1m56.804s
user 1m7.888s
sys 0m5.556s
chrism@ajax:/data/vhost/popit.staging.mysociety.org/popit$
struan
commented
10 years ago
These changes appear to make the reindexing work without timing out. I think the main issue was that all four models we're being reindexed at once, so they were all treading on each others toes. By using
mapSeriesthey run one model at a time, which seems to reduce load (though memory usage is still quite high during the reindex?).