Investigate ingest-api slow response with `/assaytype/<id>`

[yuanzhou]

We've received a few alerts from the monitoring service indicating the ingest-api service was down for a few minutes, and it has happened a few times in the past two days. Juan noticed 499 from nginx access log:

63.143.42.242 - - [30/Oct/2024:17:20:18 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:22:27 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:24:26 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:26:48 +0000] "GET / HTTP/1.1" 499 0 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:27:39 +0000] "GET / HTTP/1.1" 499 0 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
52.60.129.180 - - [30/Oct/2024:17:28:57 +0000] "GET / HTTP/1.1" 499 0 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
46.137.190.132 - - [30/Oct/2024:17:28:58 +0000] "GET / HTTP/1.1" 499 0 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:30:18 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:32:18 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"
63.143.42.242 - - [30/Oct/2024:17:34:18 +0000] "GET / HTTP/1.1" 200 43 "https://ingest.api.hubmapconsortium.org/" "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)"

My quick test shows that the ingest-api became slow to response when there are calls made against /assaytype/<id> from portal-ui.

Previously this soft-assay endpoint was only called during reindex time. According to John:

We have a hook that makes a few requests to the soft-assay endpoint on the datasets page. We could potentially add more of the soft-assay response to the dataset documents at index time to avoid this.

[yuanzhou]

The /assaytype/<id> calls entity-api first then builds the metadata based on ingest_metadata on the fly, and then the rule chain comes into play with initialization and lookup. A given entity-api response is cached for every two hours, so the final result of the soft assay is deterministic.

[yuanzhou]

Will see if this makes a difference. Still wondering why we didn't have this issue before.