WordPress / openverse

Openverse is a search engine for openly-licensed media. This monorepo includes all application code.
https://openverse.org
MIT License
244 stars 195 forks source link

Increase in timeouts in requests to Photon on thumbnail route #1231

Closed sarayourfriend closed 1 year ago

sarayourfriend commented 1 year ago

Sentry link

https://openverse.sentry.io/share/issue/f75ef09673f04c86a7ba2c358be36f78/

Description

Over the last 4 days we've gone from 0 timeouts on requests to Photon to 344. I'm reaching out to the folks that run Photon to see if they can identify the issue.

It appears to affect many providers. I extracted the IDs from the thumbnail requests that had a timeout and pulled the provider for those results:

+-----------------+
| provider        |
|-----------------|
| digitaltmuseum  |
| europeana       |
| finnishmuseums  |
| flickr          |
| geographorguk   |
| met             |
| museumsvictoria |
| nappy           |
| rawpixel        |
| rijksmuseum     |
| smk             |
| svgsilh         |
| wikimedia       |
+-----------------+

I can retrieve the upstream images just fine for the ones I tried, I suspect it is a genuine timeout. I'm in Australia, so everything on the internet is inherently slower for me, but it did take some time to retrieve some of the results I tried to hit directly, though not all.

This is reproducible: https://api.openverse.engineering/v1/images/db4e4787-5893-44cf-ae3f-b193aff4d0d5/ consistently returns a 429 reporting a timeout with Photon.

Additional context

I've marked this as high rather than critical because while it is consistent, it's still a relatively small number of requests that fail (less than 400 in 24 hours, so far).

sarayourfriend commented 1 year ago

In the meantime we can mitigate this by increasing our timeout in photon.py, though I'm not sure to what exactly.

sarayourfriend commented 1 year ago

Another issue in Sentry that seems related: https://openverse.sentry.io/share/issue/92651a89547f4e48896a9d9b25737b6f/

sarayourfriend commented 1 year ago

This should be resolved by #2426