search

openfoodfoundation / openfoodnetwork

Connect suppliers, distributors and consumers to trade local produce.
https://www.openfoodnetwork.org
GNU Affero General Public License v3.0
1.1k stars 714 forks source link

Reduce response time shop pages #11992

Open mkllnk opened 8 months ago

mkllnk commented 8 months ago

What we should change and why (this is tech debt)

According to New Relic, shopfront load times are usually okay (730 ms). The "slowest trace" is 31 s though and the longest "transaction duration" is 2 minutes. I don't know why these numbers are not the same. Another view states 3 min as maximum transaction time. Screenshot from 2024-01-03 15-34-11

Context

Impact and timeline

mkllnk commented 8 months ago

I suspected that we were maybe re-rendering product images for the shopfront but looking deeper into New Relic it looks like an n+1 problem. Certain database queries are run hundreds of times. It's not easy to see for me though what the real impact is. It may be better to experiment locally.

bmd08a1 commented 8 months ago

let me work on this. Just to confirm, shopfront page means this api app/controllers/api/v0/shops_controller.rb, right?

mkllnk commented 8 months ago

We have the shop list (index) and the individual shop (show) pages. Both need optimising, actually. I don't know where the slow spot it, though. You may need to generate some test data locally to have 1000 shops or 1000 products so see the impact.

bmd08a1 commented 8 months ago

Oh I saw the other issue on shops index too, I'll work with this shopfront page first. Can you share me some logs/traces on New Relic for this page, that could help a lot in debugging the issue

mkllnk commented 8 months ago

Can you share me some logs/traces on New Relic for this page, that could help a lot in debugging the issue

Unfortunately, I didn't find anything useful in New Relic. Also revisiting the shop didn't take that long. So it may be a caching issue. I don't know. I think that you need to find a way to re-produce it.

Matt-Yorkley commented 5 months ago

If the response time on the shop pages is normally ~700ms, those anomalous 2-minute response times are likely taking place during a period where the server is under heavy load due to something other than that page (most likely reports).