search

coopcycle / coopcycle-web

Logistics & marketplace platform. Only for worker-owned business.
https://coopcycle.org
Other
571 stars 130 forks source link

Improve tasks exports performance #4673

Closed r0xsh closed 1 month ago

r0xsh commented 1 month ago

Should rewrite tasks export mechanism à la RestaurantStat::class

Current implementation take ~ 15 minutes to export a single month for a single coop

alexsegura commented 1 month ago

Hmmm the RestaurantStats class is super optimized but super messy, not sure it's a good idea to go this route.

The task export was rewritten using Cube in f7795bde722803a4211733f58731bbe68681011f, because the previous implementation was loading objects in memory, and you had timeouts when exporting above a certain range of dates.

It's performing one big query, so I think first we could optimize that query. I think the bottlenecks lie along those lines:

It's written in a simple way in the Cube, but I think it generates an overly complex and suboptimized query in the end.

alexsegura commented 1 month ago

IMHO the way to go is #3177

If exports are managed asynchronously, they can be dead simple and just load all entities in memory.