Note that this is somewhat risky. When we did this on ubuntu.com, it locked up the database
and had to be reverted.
This appears to be because we had used up all 100 connections to the database. Our understanding
is that this is because each worker creates a separate database connection and ubuntu.com
consists of 35 pods in production and 19 pods in staging (54 total). So when we released the change
to expand each of these pods from running 1 worker to 3 workers, we would have added
162 potential extra database connections.
By contrast, we only run 5 pods of this project in production, and 3 in staging (8 total).
So if we expand our workers by a factor of three, this should only add a maximum of 16
extra database connections, which it should be able to handle.
So hopefully this change is safe.
Also patch psycopg2 with psycogreen, as we tried in ubuntu.com. Although we rolled this back in ubuntu.com out of an abundance of caution, we don't think this actually caused any issues, so it should be safe to re-do here.
Fixes #59 (at least, in every way I currently know how)
QA
dotrun
Check the API basically runs. Beyond that, I don't think we can QA much until it gets onto staging and production unfortunately.
I'm doing this now because in trying to test https://github.com/canonical-web-and-design/ubuntu.com/pull/11249, I found the security API returning 504 errors quite a lot at e.g. https://ubuntu.com/security/cves/CVE-2022-23639.json. To be able to properly test & release that PR I think we need to get this security API into a stable and responsive state.
For the rationale of why we chose 3, see https://github.com/canonical-web-and-design/ubuntu.com/pull/11212.
Note that this is somewhat risky. When we did this on ubuntu.com, it locked up the database and had to be reverted.
This appears to be because we had used up all 100 connections to the database. Our understanding is that this is because each worker creates a separate database connection and ubuntu.com consists of 35 pods in production and 19 pods in staging (54 total). So when we released the change to expand each of these pods from running 1 worker to 3 workers, we would have added 162 potential extra database connections.
By contrast, we only run 5 pods of this project in production, and 3 in staging (8 total). So if we expand our workers by a factor of three, this should only add a maximum of 16 extra database connections, which it should be able to handle.
So hopefully this change is safe.
Also patch psycopg2 with psycogreen, as we tried in ubuntu.com. Although we rolled this back in ubuntu.com out of an abundance of caution, we don't think this actually caused any issues, so it should be safe to re-do here.
Fixes #59 (at least, in every way I currently know how)
QA
Check the API basically runs. Beyond that, I don't think we can QA much until it gets onto staging and production unfortunately.