search

codeforIATI / iati-datastore

An open-source datastore for IATI data with RESTful web API providing XML, JSON, CSV plus ETL tools
https://datastore.codeforiati.org
Other
1 stars 1 forks source link

Transaction download: Eager load joined tables #347

Closed odscjames closed 2 years ago

odscjames commented 2 years ago

Performance boost to /api/1/access/transaction.csv

https://github.com/codeforIATI/iati-datastore/issues/346

odscjames commented 2 years ago

Takes downloading all GB's transactions in a CSV 20 times in 5 concurrent requests from 329 seconds to 133 seconds :cake:

markbrough commented 2 years ago

Nice! This looks good to me. Are there any significant issues this creates, e.g. in terms of significantly increasing memory usage? If not, great if you can go ahead with doing the same for the other endpoints (transactions_by_country(), etc)

odscjames commented 2 years ago

No signifiant memory drop seen on dev server in last hour when I was testing. And this is basically the same as we've done before for something else and we haven't seen any problems there.

Do you want to approve this one?

I've already commented on the issue - if you want to put remarks about further work there?