search

openaq / openaq-api-v2

OpenAQ API
https://api.openaq.org
39 stars 9 forks source link

db.py refactor for variable working memory #281

Closed russbiggs closed 9 months ago

russbiggs commented 9 months ago

to support more intesive queries we'll want to refactor db.py a bit to allow PostGres working memory to be configurable on an endpoint by endpoint basis. More complex endpoint will be allocated more working memory, less complex will get the default

caparker commented 9 months ago

I ended up adding a transaction to the db fetch method as it was the only way to get the set_config method to stick until the next query. Without the transaction asynpg will auto-commit everything and therefor the local setting (making sure its only set locally) doesnt work.

I assume that having that transaction overhead wont be an issue with all the other queries but if we see that it is we can update it so that only the endpoints that need to change settings will start the transaction.

You can see the changes here. Take a look at let me know if you see anything obvious that should be fixed. https://github.com/openaq/openaq-api-v2/pull/282/files