Catalog sync fails, running out of memory

dotmailer / dotmailer-magento2-extension

The official Dotdigital for Magento2 extension

https://dotdigital.com/integrations/magento

MIT License

48 stars 63 forks source link

Catalog sync fails, running out of memory #533

Closed scottsb closed 5 years ago

scottsb commented 5 years ago

Catalog sync fails to even get started with this error in the messages column of the cron_schedule table for the ddg_automation_catalog_sync job (actual web root snipped):

PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in [webroot]/vendor/magento/zendframework1/library/Zend/Db/Statement/Pdo.php on line 291
PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in [webroot]/vendor/colinmollenhour/credis/Client.php on line 0
PHP Warning:  Unknown: Cannot call session save handler in a recursive manner in Unknown on line 0
PHP Fatal error:  Allowed memory size of 1073741824 bytes exhausted (tried to allocate 20480 bytes) in Unknown on line 0

This is on Magento 2.2.7, using the bundled extension.

scottsb commented 5 years ago

Let me know what more information I can provide. I imagine this error alone isn't very useful, but I don't know what would help.

simon-letch commented 5 years ago

Hey @scottsb,

It would be good if you could share some information regarding catalog size, number of websites/stores and the infrastructure your Magento instance is running on. If you don't want to share this publically, please raise a support ticket with support@dotdigital.com referencing this issue and we will take a look.

As a potential workaround you can try reducing the "Transactional Data Sync Size" from the Developer section in our extension.

scottsb commented 5 years ago

@simon-letch Thanks for replying. Don't mind sharing this publicly; I'll just keep the specific website I'm discussing anonymous here.

just over 29k simple products
single website/store
cron job runs on an EC2 server dedicated to admin & cron (but really this doesn't matter, since it's hitting the configured 1GB PHP CLI process memory limit, not the server's overall memory)

scottsb commented 5 years ago

Also, I was wrong when I said this was on Magento 2.3.0. I forgot we haven't released that upgrade yet. This is currently still on Magento 2.2.7.

The "Transactional Data Sync Size" is currently set to 50, the smallest option.

simon-letch commented 5 years ago

@scottsb we believe this commit will solve your current issue with the catalog sync: https://github.com/dotmailer/dotmailer-magento2-extension/commit/f1994eaa8b8ec07649dc6239ac93157aabd3f180

We've run tests with Magento's performance toolkit datasets and found massive improvements. We were able to reproduce the out of memory exception with the medium size dataset. This was resolved after the change had been committed. Stats are in the commit message.

We can't currently release due to being mid way through Magento 2.3.1 regression cycle. Once that's been released we'll release this change to master. I'll close the issue at that point.

scottsb commented 5 years ago

Thanks. We applied this as a patch using cweagans/composer-patches in the mean time. Seems to have resolved the issue. The sync has at completed its first batch without issue.

sta1r commented 5 years ago

Fixed in 3.2.0.