Performance of additional_data module on datastore live - Githubissues

ThreeSixtyGiving / datastore

A Data Store application for 360Giving

GNU Affero General Public License v3.0

0 stars 1 forks source link

Performance of additional_data module on datastore live #130

Open michaelwood opened 1 year ago

michaelwood commented 1 year ago

Writing the additional_data for the grants takes about ~hour. This makes testing and development quite slow.

time ./datastore/manage.py rewrite_additional_data latest

real    56m5.338s
user    29m44.645s
sys     2m19.053s

Possible solutions to investigate

Parallel processing
Bulk updates to avoid hitting the db
An in-memory store such as redis
Increase the ram to match the number of CPUs