ddg_automation_cleaner is too slow

dotmailer / dotmailer-magento2-extension

The official Dotdigital for Magento2 extension

https://dotdigital.com/integrations/magento

MIT License

48 stars 64 forks source link

ddg_automation_cleaner is too slow #625

Closed mpchadwick closed 5 months ago

mpchadwick commented 6 months ago

Hi - On one of our client projects I checked today and their email_importer table had grown to be over 40GB.

I'd like to be able to more aggressively clean the table, but 30 retention duration is hardcoded.

This module should

Make the retention duration user configurable
Run the clean up job more frequently. I'd suggest running it nightly.

sta1r commented 6 months ago

@mpchadwick I agree, this has been in our backlog for a while now. We're going to sort it in our next quarterly release.

mpchadwick commented 6 months ago

@sta1r when do you foresee that being released?

sta1r commented 6 months ago

Hi, we'll be releasing 08cbbf0ee31ee66d1492a8bd51ebeaaa60e247d2 on 13/14 Feb.

mpchadwick commented 5 months ago

Thanks - Looks good although I'm not sure why you decided to require table_cleaner_interval to be set only at the CLI?

sta1r commented 5 months ago

Good question. I think with some of our configs we assume they are really developer-only switches - plus in this case, bringing the interval down can be a destructive measure. Also, adding this extra select in the system UI would probably have merited its own separate sub-section, just for the cleaner (or at least some decisions to be made regarding that).

So it will be CLI-only to start with, but I'll add a task to surface it in the UI.