Bulk Importer not importing, because 1 of our websites has dotmailer disabled

[gpcrocker]

Environment Our Magento2 instance has a more than 1 website and Dotmailer is disabled for one of them, but enabled for the other.

Problem The Bulk Importer was not importing anything for a very long time and we recently prioritized this. We discovered that the bulk importer would process 5 customer csv files from a website where Dotmailer is disabled, skip it and do nothing about it. On the next Cron run, it would repeat this exact process and never process customers csv files from our Enabled website.

Replication Reading the code specified in Technical notes should be sufficient, but to replicate, one may generate customers over a 5 day period on the dotmailer disabled website, on the next 5 days generate customers on the enabled website. The expected result is that the 5 days worth of customers on dotmailer enabled website will never be processed as it will be stuck on 5 days worth of customer on dotmailer disabled website.

Temporary Solution To fix this and get things rolling, we increase the Bulk limit to an arbitrary figure. This worked and in fact your APIs blocked us temporarily for making too many API calls as we had a lot of files to send.

Technical notes Function vendor/dotmailer/dotmailer-magento2-extension/Model/Importer.php:processQueue() will only process the 5 customer csv files of the disabled and will be persistantly ignore them at vendor/dotmailer/dotmailer-magento2-extension/Model/Sync/Contact/Bulk.php:67

Hotfixed our Magento2 with vendor/dotmailer/dotmailer-magento2-extension/Model/Importer.php:231 $this->bulkSyncLimit = 393; //arbitrary figure

Permenant Solution A solution would be to amened the way the collection is gathered through Importer.php:242, because the SQL it executes is

SELECTmain_table.* FROMemail_importerASmain_tableWHERE (((import_type= 'Contact') OR (import_type= 'Guest') OR (import_type= 'Subscriber'))) AND (import_mode= 'Bulk') AND (import_status= 0)

But it should be more like :

SELECTmain_table.* FROMemail_importerASmain_tableWHERE (((import_type= 'Contact') OR (import_type= 'Guest') OR (import_type= 'Subscriber'))) AND (import_mode= 'Bulk') AND (import_status= 0) andwebsite_idNOT IN (select scope_id aswebsite_idfromcore_config_datawherepathlike 'connector_api_credentials/api/enabled' andvalue= 0)

[sta1r]

@gpcrocker Thanks for raising this. Just to let you know that we've worked on a fix for this, which is currently in review. We'll update this ticket after it's released.

[sta1r]

@gpcrocker Fix released in v3.4.0.