search

PrestaShopCorp / ps_eventbus

PS EventBus is the CloudSync companion which paves the way for cloud synchronization
https://prestashop.fr/built-for-prestashop/
Open Software License 3.0
6 stars 7 forks source link

eventbus_incremental_sync table constantly grows with millions of lines #274

Closed jokesterfr closed 7 months ago

jokesterfr commented 7 months ago

Reported from the PrestaShop open source slack by Daniel : https://prestashop.slack.com/team/U0135GYKH0C

Feedback from Daniel:

It has a table eventbus_incremental_sync that just keeps adding milions of rows.. Some kind of cleanup is needed here.

Answer (for what's working and what's left to do): 

Each time an action is performed on the shop, some hooks are triggered to fullfil the outbox. And each time the CloudSync processes ask the shop, the data is sent back.
However if a shop has never been synchronized with CloudSync, the incremental mode should never be used, and the hooks handles be voided. Until the full sync ends (if ever called, this depends on the merchant onboarding action).
I see another pattern that could be developed right away, is the synchronization failure or the merchant consented to share "carts" and then revoked to sync its carts later on. That would prevent calls from CloudSync to unstack messages from the outbox, and this last one would continue to grow as time goes.
A fix would be to reset the incremental sync for some shopContents (also called collections) if the outbox exceeds a limit. I can get back to you once we have this new protection live, Github issues are open if you need anything else.

I will reach the team and get back to you soon,

Prestaworks commented 7 months ago

Here is an example from a live site bild

it's about 4,5 million rows and keeps growing. the oldest entry is from 2023-09-05

My suggestion here is a cron file that can be used to clean this table and maybe keep data 3 months back? If the merchant don't want to clean up, they can just skip to activate the cronjob, what do you think?

jokesterfr commented 7 months ago

Dear @Prestaworks,

thanks to @fox-john work, we have a patch for you (see #276), these are scripts you can run to clear unwanted data from your database. This will still be compliant with the ps_eventbus normal behaviour, and wont mess with the ps_eventbus/CloudSync nominal bahaviour.

These MySQL scripts can now be found here: https://github.com/PrestaShopCorp/ps_eventbus/tree/main/sql/tools

By the way, 2 long term fixes will soon come. Our internal references are:

best,