email_import shows the error "Error unknown" for orders batch exporting

[nans]

Hello, we have an issue with the infinity loop because of an empty value transactional_data_break

No default value https://github.com/dotmailer/dotmailer-magento2-extension/blob/master/etc/config.xml#L295

Not visible in the admin panel: https://github.com/dotmailer/dotmailer-magento2-extension/blob/master/etc/adminhtml/system.xml#L1752

What is the problem: We have a lot of orders and at some point the email_importer table began to overflow. Due to the endless selection of orders, the module tried to place all the data in the import_data column, but due to the 16MB limit, not all of the data was saved.

What we did to solve the problem: The value (50) for the "transactional_data_break" attribute was set through the database.

Sample:

[sta1r]

@nans Hello, sorry for the delay on this.

The config key that governs how many orders to include in one batch, to be sent to the importer table, is:

connector_developer_settings/import_settings/mega_batch_size_orders

The default for this is 5000. I guess this results in too much JSON data for you, perhaps because you are syncing many attributes? And/or, your orders typically contain many products?

Amending transactional_data_break will resolve the issue too, but this is more drastic, because it means the sync will not run continuously, and will stop after 50 records, not starting again until the next cron execution (by default every 15 minutes). So, orders will be pushed at a much slower rate.

Incidentally, if you haven't already, you should increase your transactional data sync size to greater than 10 (certainly if you end up removing transactional_data_break) because this does more, smaller, loops to make a mega batch - which is less efficient.

[sta1r]

I'm going to reduce the default mega batch size for orders to 4000 in our next release.

In the meantime, a reminder that you can tune the size of the import data for orders sync using for example:

bin/magento config:set connector_developer_settings/import_settings/mega_batch_size_orders 2000