intermittent timeouts or Gateway timeout errors from Security API routes for specific cves/{cve_id}.json or notices/{usn_id}.json

Summary

[Please describe the issue.] From a remote client perspective, we are seeing fairly frequent timeouts either (Gateway timeout error 504) of just a 30 second lag in responses from the Security API when performing GETs via either curl or python requests to either of the routes u.com/security/cves/{cve_id}.json or u.com/security/notices/{notice_id}.json.

Process

repeated curl attempts over a period of time seem to hit this issue (pointing potentially to an HA Proxy round-robing config type issue maybe)?

curl -X GET "https://ubuntu.com/security/notices.json?details=USN-4510-1" -H "accept: application/json"
 curl https://ubuntu.com/security/notices/USN-4559-1.json -H "accept: application/json" > usn.3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100  1444  100  1444    0     0     46      0  0:00:31  0:00:30  0:00:01   338

The network timeouts talking to the security API seem to hit our integration tests failly frequently too (which leverages python requests for GETs against the Security API : https://github.com/canonical/ubuntu-advantage-client/issues/1374

Current and expected result

It seems strange to have a 30 second timeout on repeated requests to the Security API. Is there something either misconfigured with an HA frontend or is the a per-client throttling limit that we are hitting when we try 3-5 requests back to back against the API that is reducing out throughput of API calls?

Browser details

N/A

canonical / ubuntu-com-security-api