Import Process in Magento 2.4.6 Interfering with Each Other (Magento 2.4.6 )

[mohammadbharmal]

Preconditions and environment

• Magento 2.4.6

• Critical Bug in Magento 2.4.6 Import Process: Interference Between Simultaneous Imports

• When two import processes are initiated simultaneously in Magento 2.4.6, a severe bug arises. In my case, during the customer composite import and product import occurring concurrently, an error occurs within the customer import process. Upon further investigation and debugging, it becomes apparent that Magento erroneously incorporates product data into the customer import procedure.

• More details and screenshot

-In Magento 2.4.6, a critical bug arises in the import process due to the default batch size limit defined in the configuration file "vendor/magento/module-import-export/etc/config.xml". By default, the batch size is set to 100. Consequently, when importing data, Magento inserts records into the "importexport_importdata" table based on this batch size. For instance, if there are 1400 records, Magento will insert 14 rows into the table during the import validation phase.

Upon clicking the import button, Magento retrieves the data from the aforementioned table and proceeds with the import process. However, an issue occurs when any import contains more than 100 records, leading to the creation of multiple entries in the table. This becomes problematic when two import processes simultaneously validate their files, as each process adds its validated data to the "importexport_importdata" table. This interference causes issues during the subsequent import steps.

Import error screenshot which is showing customer import is taking product data,

Import history screenshot which is showing two import has triggered at same time and it's import one data for both different import

Config file

Steps to reproduce

Steps:-

1) Log in to Magento 2.4.6 admin in two separate system. 2) In both systems, navigate to "System" > "Data Transfer" > "Import". 3) In the first system, select "Customers and Addresses" (single file) with billing and shipping address data, ensuring there are more than 500 records. 4) In the second system, select "Catalog Product Import" with 50 to 60 records. 5) Validate both selected imports. 6) In the first system, click on the "Import" button. 7) Within the next second, click on the "Import" button in the second system.

Expected result

Both imports should be executed without any errors or interference between them.

Actual result

Throwing error in one import

Additional information

No response

Release note

No response

Triage and priority

• [ ] Severity: S0 - Affects critical data or functionality and leaves users without workaround.
• [ ] Severity: S1 - Affects critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S2 - Affects non-critical data or functionality and forces users to employ a workaround.
• [ ] Severity: S3 - Affects non-critical data or functionality and does not force users to employ a workaround.
• [ ] Severity: S4 - Affects aesthetics, professional look and feel, “quality” or “usability”.

[m2-assistant[bot]]

Hi @mohammadbharmal. Thank you for your report. To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce. To deploy vanilla Magento instance on our environment, Add a comment to the issue:

• @magento give me 2.4-develop instance - upcoming 2.4.x release
• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this

• To learn more about issue processing workflow, refer to the Code Contributions.

  Join Magento Community Engineering Slack and ask your questions in #github channel. :warning: According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting. :clock10: You can find the schedule on the Magento Community Calendar page. :telephone_receiver: The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[m2-assistant[bot]]

Hi @mohammadbharmal! :wave: Thank you for collaboration. Only members of Community Contributors Team are allowed to be assigned to the issue. Please use @magento add to contributors team command to join Contributors team.

[mohammadbharmal]

@magento add to contributors team

[m2-assistant[bot]]

Hi @mohammadbharmal! :wave: Thank you for joining. Please accept team invitation :point_right: here :point_left: and add your comment one more time.

[m2-assistant[bot]]

Hi @engcom-Bravo. Thank you for working on this issue. In order to make sure that issue has enough information and ready for development, please read and check the following instruction: :point_down:

• [ ] 1. Verify that issue has all the required information. (Preconditions, Steps to reproduce, Expected result, Actual result).
• [ ] 2. Verify that issue has a meaningful description and provides enough information to reproduce the issue.
• [ ] 3. Add Area: XXXXX label to the ticket, indicating the functional areas it may be related to.
• [ ] 4. Verify that the issue is reproducible on 2.4-develop branch
  Details

  - Add the comment @magento give me 2.4-develop instance to deploy test instance on Magento infrastructure.
  - If the issue is reproducible on 2.4-develop branch, please, add the label Reproduced on 2.4.x.
  - If the issue is not reproducible, add your comment that issue is not reproducible and close the issue and stop verification process here!
• [ ] 5. Add label Issue: Confirmed once verification is complete.
• [ ] 6. Make sure that automatic system confirms that report has been added to the backlog.

[Thundar]

The same problem occurs even with two different admin users in parallel. All imports are saved on importexport_ table, and then the table with different entities stored is loaded by each separated process. The older process finds wrong column names, from the younger, and dies.

[mohammadbharmal]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @mohammadbharmal. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @mohammadbharmal, here is your Magento Instance: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering Admin access: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering/admin_d491 Login: ff6ecb38 Password: b77db3dc453a

[mohammadbharmal]

@magento To clarify, it seems that the provided instance has a built-in error that prevents the successful import of the sample customer import with dummy user data.

Below is the CSV file which I am trying to import.

new_customer.csv

[mohammadbharmal]

@magento give me 2.4.6-develop instance

[magento-deployment-service[bot]]

Hi @mohammadbharmal. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @mohammadbharmal, unfortunately there is no ability to deploy Magento instance at the moment. Please try again later.

[mohammadbharmal]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @mohammadbharmal. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @mohammadbharmal, here is your Magento Instance: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering Admin access: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering/admin_0b5e Login: 8917120e Password: a7253f7bf47c

[mohammadbharmal]

Reproduced on 2.4.x

Issue is confirmed

[mohammadbharmal]

@engcom-Bravo issue confirmed on 2.4-develop instance

[engcom-Bravo]

@magento give me 2.4-develop instance

[magento-deployment-service[bot]]

Hi @engcom-Bravo. Thank you for your request. I'm working on Magento instance for you.

[magento-deployment-service[bot]]

Hi @engcom-Bravo, here is your Magento Instance: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering Admin access: https://75989f09435aca0ef6168b9f45f29849.instances-prod.magento-community.engineering/admin_453e Login: 2998055a Password: ca70b5df36c1

[mohammadbharmal]

@engcom-Bravo, Below is the sample csv files which I have prepared and tested with Magento develop instance,

Customer composite import file

customer_composite (4).csv

Product sample import file

catalog_product (2).csv

[engcom-Bravo]

Hi @mohammadbharmal,

Thank you for reporting and collaboration.

Verified the issue on Magento 2.4-develop instance and the issue is reproducible.Kindly refer the screenshots.

Steps to reproduce

• Log in to Magento 2.4.6 admin in two separate system.
• In both systems, navigate to "System" > "Data Transfer" > "Import".
• In the first system, select "Customers and Addresses" (single file) with billing and shipping address data, ensuring there are more than 500 records.
• In the second system, select "Catalog Product Import" with 50 to 60 records.
• Validate both selected imports.
• In the first system, click on the "Import" button.
• Within the next second, click on the "Import" button in the second system.

For one import file is valid but still we are getting error and other one we are able to import successfully.

Hence Confirming this issue.

Thanks.

[github-jira-sync-bot]

:white_check_mark: Jira issue https://jira.corp.adobe.com/browse/AC-8997 is successfully created for this GitHub issue.

[m2-assistant[bot]]

:white_check_mark: Confirmed by @engcom-Bravo. Thank you for verifying the issue.
Issue Available: @engcom-Bravo, You will be automatically unassigned. Contributors/Maintainers can claim this issue to continue. To reclaim and continue work, reassign the ticket to yourself.

[ga3003]

After debugging for some time, I found the below pattern of the issue. So whenever we upload a file with more than 100 rows, Magento divides the file into multiple bunches and gets stored in the importexport_importdata database table. Now when execution starts, these bunches are conflicting and causing errors. I have tried multiple times and its the same result every time I upload files with more than 100 rows.

Please let me know, if anyone have solved this temporarily or permanently.

[mohammadbharmal]

@ga3003 ,

https://magento.stackexchange.com/questions/367882/import-process-in-magento-2-4-6-interfering-with-each-other-magento-2-4-6/367884#367884

Try above solution as temporarily fix

[ga3003]

Thanks, @mohammadbharmal.

But what I have done is something similar to what you have mentioned, I have increased this batch size to 300 so that we should not face a timeout error. And asked our catalog team to divide the files into chunks of 300. That makes our work more fast and easy. Hope this will help others to work temporarily with this issue.

[mohammadbharmal]

@ga3003 ,

Yah, that exactly I have mentioned in the stackexchange answer.

[ConradoSM]

the error persists even after applying the temporary fix:

[mohammadbharmal]

@ConradoSM

it's looks like importexport_importdata table has some old record due to error, truncate that table and try to import again, it's should work.

[ConradoSM]

@mohammadbharmal I attached a video

https://github.com/magento/magento2/assets/44508579/7a935d86-c0e4-4957-a66f-7e758264ab3c

[mohammadbharmal]

@ConradoSM, yes, that is the problem.

To address this issue, I have implemented an additional solution in my flow, considering that I use a third-party ERP for importing data. The steps are as follows:

Configure the ERP import to be synchronized, ensuring that the process runs in a controlled and orderly manner.

Before starting the validation process, remove the entry from the importexport_importdata table. You can achieve this by using a plugin or a similar mechanism.

By applying the above flow, I have implemented a temporary solution to handle the conflict and ensure smoother data imports.

[ConradoSM]

@mohammadbharmal the problem with the errors in my case is in my cloud environment with the slave mysql instance, we reported with the support team and a workaround they send me a configuration for the .magento.env.yaml file:

[mohammadbharmal]

@ConradoSM , is it working for you?

[jeremycharron]

We had multiple projects on 2.4.6 with similar product import issues and missing attributes. The solution suggested by @ConradoSM and also recommended by our Adobe Cloud rep fixed our problems.

Raw code for SEO :

DATABASE_CONFIGURATION: 
      _merge: true 
      slave_connection: 
        default: 
          synchronous_replication: false

[mabaud]

The issue is caused by this commit : https://github.com/magento/magento2/commit/c1b596e78bd454ada020b4cdba801b73c51e19ad#diff-c25683ebd4aa61ea427c0f51d5a95410ca67c86f6738bdbbfe908753f34d055b

In M2.4.5, Magento used to clear the table importexport_importdata before adding new bunches. Now in 2.4.6, Magento only clear rows processed. The problem is that Magento iterates on the importexport_importdata table without checking the entity of the rows retrieved, causing this bug : https://github.com/magento/magento2/blob/6cc0d28cf66074adebc261e981eb35811601f813/app/code/Magento/ImportExport/Model/ResourceModel/Import/Data.php#L65

So, in the backoffice, if you :

• Check the data of a stock file for instance (System > Import), it will add rows in the importexport_importdata table
• Then check the data of a customer file (System > Import), it will also add rows in the importexport_importdata table
• Then click on import, Magento will try to import the bunches corresponding to the stock file instead of retrieving bunches of the customer file.

Quick fix for that issue : replace that line https://github.com/magento/magento2/blob/6cc0d28cf66074adebc261e981eb35811601f813/app/code/Magento/ImportExport/Model/Import/AbstractEntity.php#L424

By $this->_dataSourceModel->cleanBunches();

That way we get back to the old 2.4.5 behavior, meaning parallel import are not possible.

The proper solution is to filter by entity when retrieving data from the importexport_importdata table.

[BeardTech]

here's the patch that i've applied on my side :

--- Model/Import/AbstractEntity.php 2023-02-23 15:11:00.000000000 +0100
+++ Model/Import/AbstractEntity.php 2024-01-16 10:53:49.738569274 +0100
@@ -421,7 +421,7 @@
         $startNewBunch = false;

         $source->rewind();
-        $this->_dataSourceModel->cleanProcessedBunches();
+        $this->_dataSourceModel->cleanBunches();
         $mainAttributeCode = $this->getMasterAttributeCode();

         while ($source->valid() || count($bunchRows) || isset($entityGroup)) {
--- Model/Import/Entity/AbstractEntity.php  2024-02-07 12:04:54.000000000 +0100
+++ Model/Import/Entity/AbstractEntity.php  2024-04-10 15:31:22.642293121 +0200
@@ -396,7 +396,7 @@
         $skuSet = [];

         $source->rewind();
-        $this->_dataSourceModel->cleanProcessedBunches();
+        $this->_dataSourceModel->cleanBunches();
         while ($source->valid() || $bunchRows) {
             if ($startNewBunch || !$source->valid()) {
                 $this->ids[] =

I've applied it with cweagans/composer-patches

thanks @mabaud for the help :)