Magento 2.2.5 (Enterprise) gives massive data corruption with configurable products and product with links

[koenner01]

Issue Brief

We have discovered a problem with the Products\ExportEntity $children data in our enterprise setup. The module is assuming that the product_id in the catalog_product_entity_link table is the actual product entity_id but this is NOT correct. The product_id is actually the sequence_product. Because of this (miss) assumption the data for products with links or products that are configurable are corrupted.

Environment

• [x] PHP Version: (7.0.27)
• [x] Magento Version: 2.2.5
• [x] Magento Enterprise: (yes)
• [x] Tweakwise Export Version: v1.1.4
• [x] Magento Deploy Mode: (production)
• [x] Third party modules: a lot

Steps to reproduce

1. Have a Magento 2.2.5 enterprise set up
2. Have at least 1 configurable product with simples
3. Check the catalog_product_super_link table and look for a link of your newly created configurable product
4. Look at the product_id of the link and make sure there is a product in your Magento setup that has the value of the product_id (=sequence_product) as it's entity_id.
5. Run the emico_tweakwise_export cron
6. Look at the data in the export

Actual result

The configurable product in the export contains faulty data of products that have their entity_id value equal to the values in the sequence_product table.

Expected result

The configurable product in the export should contain the data of the simples

The \Emico\TweakwiseExport\Model\Write\Products\CollectionDecorator\Children contains several methods that contains raw sql queries. For example the addConfigurableChildren function and the function addLinkChildren assumes that the product_id is the actual entity_id of the product (which it's not).

[edwinljacobs]

Hello,

We added a fix for this issue, the problem is indeed that the sequence_product table is referenced in the various link tables. This differs in the community edition and remained unnoticed in our enterprise demo store since the ids in the sequence_product tables matched exactly with the entity ids of the corresponding products. Furthermore, at the time the export was developed this reference was still pointing at the catalog_product_entity table.

How to implement? We created a new branch "test" and the fix is merged in this branch (it is not yet implemented in the master branch or in any tag) you can use the test branch by requiring emico/tweakwise-export "dev-test" using composer. We just pushed the fix to the test branch so that this is available for you but note that this is not yet final as we are still testing.

With kind regards, Edwin Jacobs

[hostep]

Thanks @edwinljacobs

I'm currently testing this, have to use "emico/tweakwise-export": "dev-test as 1.1.4", in our composer.json though, since there is a version constraint in the tweakwise module which would otherwise conflict with dev-test. But that's not very important.

At first look, it looks ok, but I'm still noticing some other strange problems where certain attribute values are incorrect. Still doing some debugging at the moment, to try to figure out where they are coming from. I can not reproduce this with only a few products in the database, it only seems to happen when you have a lot of products in the database, might be something batch related, where the state is shared from one batch to the other where the code assumes it's not shared, but that's only guessing here.

I'll try to keep you updated with further findings.

[hostep]

Hi @edwinljacobs

Ok, ignore the comment above, it has nothing to do with a shared state or something like that.

There are more queries incorrect I believe. Especially in Emico\TweakwiseExport\Model\Write\Products\CollectionDecorator\Children::addConfigurableChildren, but most likely also in the addBundleChildren and addLinkChildren methods.

In our case, it looks like the $parentIds array contains the wrong id's, it contains the entity_id instead of the row_id, so it uses the entity_id in the query which runs against the catalog_product_super_link on the parent_id column, which maps to the row_id column in the catalog_product_entity table. This then in turns fetches the wrong children.

Can you double check if my findings are correct? And if you can confirm, can you maybe figure out a solution for this problem as well?

Thanks!

[edwinljacobs]

Yeah thats problably the issue.

Here is the most recent commit merged to test https://github.com/EmicoEcommerce/Magento2TweakwiseExport/commit/3aff8eea686982a1daeaf8ab0c401ea6a31db765

In https://github.com/EmicoEcommerce/Magento2TweakwiseExport/blob/9639afa397e60a401c0c4b54dd397b492e9f1984/src/Model/Write/EavIterator.php#L287-L356 Is responsible for creating the original select object for the type of object that is being exported (i.e. products and categories) as you pointed out we dont differentiate between the case enterprise where the identifier field in use is 'row_id', and community where the identifier field is 'entity_id'. This should be fixed with the above commit.

We currently investigating if for example the price decorator or the Category reference decorator is also affected by the same issue. If this is the case we will also fix those and update our findings in this thread.

As a side note: The method calls to check if we are dealing with enterprise or community feel incorrect from an architecture standpoint, so it could be that we will overhaul the export architecture at some point in the future.

[hostep]

Thanks again!

Will test tomorrow and let you know the results.

As a side note: The method calls to check if we are dealing with enterprise or community feel incorrect from an architecture standpoint, so it could be that we will overhaul the export architecture at some point in the future.

Indeed, just don't use raw SQL anymore, use function calls to obtain the info you need. And let core Magento code deal with if it is an Enterprise or Community version.

Maybe also review the complexity of this module, because the way it is organised is very clean and cleverly done, but is just not very well suited for just exporting a bunch of data.

The reorganisation should then hopefully also tackle the huge memory consumption problem (we had to increase the PHP memory_limit to 3GB today just for this export to be able to run with a production database, which is kind of insane)

[hostep]

Hi @edwinljacobs

Tested again with latest test branch, but now we get an exception:

  [Emico\TweakwiseExport\Exception\InvalidArgumentException]
  Could not find export entity with id 85065

Exception trace:
 () at vendor/emico/tweakwise-export/src/Model/Write/Products/Collection.php:84
 Emico\TweakwiseExport\Model\Write\Products\Collection->get() at vendor/emico/tweakwise-export/src/Model/Write/Products/CollectionDecorator/Children.php:271
 Emico\TweakwiseExport\Model\Write\Products\CollectionDecorator\Children->loadChildAttributes() at vendor/emico/tweakwise-export/src/Model/Write/Products/CollectionDecorator/Children.php:100
 Emico\TweakwiseExport\Model\Write\Products\CollectionDecorator\Children->decorate() at vendor/emico/tweakwise-export/src/Model/Write/Products/Iterator.php:107
 Emico\TweakwiseExport\Model\Write\Products\Iterator->processBatch() at vendor/emico/tweakwise-export/src/Model/Write/Products/Iterator.php:94
 Emico\TweakwiseExport\Model\Write\Products\Iterator->getIterator() at vendor/emico/tweakwise-export/src/Model/Write/Products.php:120
 Emico\TweakwiseExport\Model\Write\Products->exportStore() at vendor/emico/tweakwise-export/src/Model/Write/Products.php:93
 Emico\TweakwiseExport\Model\Write\Products->write() at vendor/emico/tweakwise-export/src/Model/Write/Writer.php:121
 Emico\TweakwiseExport\Model\Write\Writer->write() at generated/code/Emico/TweakwiseExport/Model/Write/Writer/Proxy.php:127
 Emico\TweakwiseExport\Model\Write\Writer\Proxy->write() at vendor/emico/tweakwise-export/src/Model/Export.php:172
 Emico\TweakwiseExport\Model\Export->Emico\TweakwiseExport\Model\{closure}() at vendor/emico/tweakwise-export/src/Model/Export.php:86
 Emico\TweakwiseExport\Model\Export->executeLocked() at vendor/emico/tweakwise-export/src/Model/Export.php:196
 Emico\TweakwiseExport\Model\Export->generateToFile() at vendor/emico/tweakwise-export/src/Cron/Export.php:62
 Emico\TweakwiseExport\Cron\Export->execute() at n/a:n/a

There is a product with row_id 85065, but not with entity_id 85065 in this case. So something is still not 100% correct I think?

Sorry for the few details, I don't have a lot of time today to put into this issue. Hope this is enough info for you to figure out what is wrong this time. If you need more info, please let me know and I'll see if I can provide it to you.

Thanks again!

[edwinljacobs]

Hi,

Thanks for the update. The reason we chose the current architecture is that M1's methods for obtaining data are very slow. At the time of development M2 was still at 2.0.x, and we where not convinced that data loading would be fast enough hence the current structure.

To get back at the exception, we will look into it as well. Do you know of a way for me to get your situation, meaning that my row ids always match with my entity ids, hence my fixes are kind of in the dark. I do not know of a consistent way to get your situation. If you know some method to get your situation that would greatly help me. I also am low on time today but will check for update regularly. Ill look into this as soon as I can.

With kind regards

[hostep]

Hi @edwinljacobs

I was thinking about that as well, and talked to the PM here who took it up with the client if it would be ok to send you our full database. Since that will probably make your live much easier to test and debug this. So I'm going to do that somewhere in the next few hours, I'll try to find your email address in the git commits of this repo and send it over there (unless you want me to send it to a specific email address?).

[edwinljacobs]

Hi,

Thanks for the quick reply, if it is possible you can send it to ejacobs@emico.nl, that would greatly help! Thanks for trying to facilitate this for us!

With kind regards

[hostep]

Hi @edwinljacobs

I send you an email last Wednesday but haven't heard back. Just checking in if you received the email I send you? (maybe it ended up in your spam?)

Thanks!

[edwinljacobs]

Hi,

I found your email, thanks for providing a database! We will get back to you soon.

With kind regards

[edwinljacobs]

Fix has been released in 1.2.0