FAC - Bug: Occasional API timeout errors citing connection pool

stucka commented 1 month ago

Describe the bug

I've got code that sometimes errors out making the API calls. I'm building in a retry functionality now, but an example: 06:39:19 ERROR:API failure: b'{"code":"PGRST003","details":null,"hint":null,"message":"Timed out acquiring connection from connection pool."}'

I verified this error has occurred more than once, and suspect it's likely happened all of the five code failures in the last five days. The script runs six times a day in the middle of the hour, so it's a not-insignificant failure rate, potentially in the ballpark of about 15%.

Steps to reproduce the bug

Appears to be intermittent. The code that generated the above error:

...
    r = requests.get(f"https://api.fac.gov/general?fac_accepted_date=gte.{asofdate}", headers=requestheaders)
    if not r.ok:
        logger.error(f"API failure: {r.content}")
...

Expected Behavior

API should be stable -- perhaps a wait-and-retry internally, or a larger connection pool.

Screenshots

No response

System setup

This was hit using Python's requests module from within a Jupyter Notebook. Code has been stable for about a year.

Additional context

I don't know if this may be linked to #4285 . I found no filed issues with "connection pool"

Code of Conduct

[X] I agree to abide by the Digital.gov Community Guidelines and the TTS Code of Conduct. Respect your peers, use plain language, and be patient.
[X] I checked the current issues for duplicate bug reports.

jadudm commented 1 month ago

So you get a quick initial response: we're still investigating, but you are correct: https://github.com/GSA-TTS/FAC/issues/4285 is related/where we're tracking the initial investigation.

stucka commented 1 month ago

Thank you. We appreciate you and all the team does.

GSA-TTS / FAC