Merge of EAV Structure of Magento 1 with Magento 2 > Product Information lost caused by missing merge.

codebase-md commented 3 years ago

Preconditions

Magento 1.9.0.1, Magento 2.3.5-p1
PHP 7.2.32
DTM 2.3.5

Steps to reproduce

Migrate settings and data from M1 to M2

Expected result

Custom attributes from M1 attribute set 'Default' attribute groups 'General' and 'Prices' are merged into M2 attribute set 'Default' groups 'General' and 'Prices'.

Actual result

Custom attributes are migrated but not merged into attribute groups 'General' and 'Prices'

Additional notes

This is shop breaking behavior since all product information regarding the not merged attributes is lost.

OldPlanets commented 3 years ago

I ran into the same problem (I think from the description). The issue seems to be https://github.com/magento/data-migration-tool/blob/7191f7c1e02256527b88de80a3d7ab86d403d3dd/src/Migration/Step/Eav/Data.php#L312: The ATTRIBUTE_SETS_NONE_DEFAULT flag removes the first attribute set from receiving the M2 default attributes. I don't know the intention behind it since I'm not entirely familiar with the code, but it's clearly a bug in my case which removing it (defaults to ATTRIBUTE_SETS_ALL) fixes it and let's me migrate my products.

p810 commented 3 years ago

@OldPlanets I removed that line but I'm now getting the following error at the EAV migration step:

SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry '4-product-details' for key 'EAV_ATTRIBUTE_GROUP_ATTRIBUTE_SET_ID_ATTRIBUTE_GROUP_CODE'

Any idea why this is happening? The migration ran fine beforehand with the exception of the missing product and product category data.

I'm running this against a clean installation of Magento 2.3.5, right after running magento setup:install. Should also note, this is with enterprise edition.

p810 commented 3 years ago

Running with the 2.3-develop branch fixed the above issue. However, I noticed that the missing category and product data issue was persisting, only this time it didn't apply to all attributes - for example, the price field was displaying correctly while product names were blank.

I dug into this a bit and confirmed that it wasn't related to the attribute set IDs like mentioned in another thread, and it wasn't because the data hadn't been migrated. After going back and forth with it for a little bit, I realized that I could change the name of a category or product in the admin panel, and that entity would then display its new name. This led me to check what the difference was between the migrated row and the newly created one. I noticed that the newly created row was not in catalog_*_entity_varchar like it should have been (and which the migrated rows belonged to), but instead, it was in catalog_*_entity_text.

I'm not sure why, but it looks like these default attributes had their backend_type changed from varchar to text. Since all of the values are migrated over to the varchar tables, the fix is as simple as updating the affected attributes. This is possible with the following query (just remember to change the asterisk to the entity type you're targeting):

update eav_attribute a set a.backend_type = 'varchar' where a.backend_type = 'text' and a.attribute_id in (
  select distinct b.attribute_id from catalog_*_entity_varchar b
);

... and then run bin/magento c:c eav and bin/magento c:f eav to flush the EAV cache. Voila, the admin panel will no longer have holes of missing data.

This behavior was observed when migrating from Magento Commerce 1.14.4.5 to 2.3.5. I also noticed it occurred with some custom customer attributes as well. I'm trying to determine whether this is a bug with Magento, the data migration tool, or specifically our installation. I'll submit a PR if it turns out to be related to the migration tool.

kanevbg commented 3 years ago

For the record: Migration from 1.9.4.3 to 2.3.5-p2 using 2.3-develop version of the data-migration tool. I have missing attribute manufacturer in all attribute sets and the data related to the products is not showing up in the backend after manually adding it to the the attribute sets.

PivitParkour94 commented 3 years ago

Getting the same issue as @kanevbg. I was going to make a separate issue for it. Error is when upgrading from M1.9.3.7 to 2.3.5-p2

Can someone point to where the mapping of the attribute codes is defined? I assume there's just a missing mapping, but it could also just be a bug in copying the attribute values.

Found this that could possibly be similar for the countryofmanufacturer attribtue

    <!-- ./vendor/magento/data-migration-tool/etc/opensource-to-opensource/class-map.xml.dist -->
    <rename>
        <from>catalog/product_attribute_source_countryofmanufacture</from>
        <to>Magento\Catalog\Model\Product\Attribute\Source\Countryofmanufacture</to>
    </rename>

PivitParkour94 commented 3 years ago

A database level check to see the data for easy testing

use M1; 
SELECT COUNT(*) FROM catalog_product_entity_int WHERE attribute_id = (SELECT attribute_id FROM eav_attribute WHERE attribute_code = 'manufacturer');
// Returns x results

use M2; 
SELECT COUNT(*) FROM catalog_product_entity_int WHERE attribute_id = (SELECT attribute_id FROM eav_attribute WHERE attribute_code = 'manufacturer');
// Returns 0 results

PivitParkour94 commented 3 years ago

I've narrowed it down to the PostProcessing step. When a migration is done without that step the manufacturer data is present on the M2 instance. Then running just that step, the manufacturer data is lost. Might be something weird like the manufacturer in this particular instance is on the default attribute set and is getting skipped or something and cleaned up as "leftover" eav data or something... I'll keep digging

PivitParkour94 commented 3 years ago

Ok so the manufacturer and color attribute values are lost for some reason. The eav_entity_attribute table (which links attributes with attribute sets) is missing the manufacturer attribute for some reason. It's not obvious why that's being skipped. There's no mention of manufacturer in all of the config mapping files or anything.

As it is not added, it shows up in the leftover attribute cleanup and the values are then removed. A quick fix is to remove the catalog_product_entity_int table from the leftover cleanup, but the underlying issue of the manufacturer not being added from the M1 default attribute set to the M2 default attribute set remains in question still. (This is just in my case)

SELECT eav_attribute.attribute_code, COUNT(eav_attribute.attribute_code) as "Leftover Attribute Count"
FROM eav_attribute,
     catalog_product_entity_int
WHERE catalog_product_entity_int.attribute_id = eav_attribute.attribute_id
  AND catalog_product_entity_int.value_id IN (
    SELECT `cpev`.`value_id`
    FROM `catalog_product_entity_int` AS `cpev`
             INNER JOIN `catalog_product_entity` AS `cpe` ON cpe.entity_id = cpev.entity_id
    WHERE (cpev.value IS NOT NULL)
      AND (cpev.attribute_id NOT IN (SELECT `eea`.`attribute_id`
                                     FROM `eav_entity_attribute` AS `eea`
                                     WHERE (eea.attribute_set_id = cpe.attribute_set_id)))
)
GROUP BY eav_attribute.attribute_code;

I used that query to figure out what attributes were being lost and get a good understanding of why.

Not too sure where to go from here, other than removing the leftover attribute cleaning, which doesn't seem like the right solution though.

Morgy93 commented 3 years ago

Same issue here with Magento 1.9.4.5 to 2.4.1 Doesn't work if attribute is in "general" tab but if I move it to something else it works.

The fun thing is that the migration worked fine prior to 2.4.1 but that doesn't match with all the reports I found.

Maybe this is also related to #812

norgeindian commented 3 years ago

I faced the same issue. Attributes in the General Tab are not migrated at all. Debugged a while an actually found a reason and solution in the end. Maybe that helps someone:

The entries in the catalog_product_entity_... tables are removed in \Migration\Step\PostProcessing\Data\AttributeSetLeftoverDataCleaner::clean() But the problem is not there, the problem is a lot earlier in the EAV step. The entries are removed, because there is no fitting entry in eav_entity_attribute for the attribute. This is caused by \Migration\Step\Eav\Data::migrateCustomEntityAttributes() Here there are all records from the M1 table taken, but some are removed during the following check: !isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']]) mapAttributeGroupIdsSourceDest is built in migrateCustomProductAttributeGroups and the interesting part happens in createMapProductAttributeGroupIds. There the $sourceKey and the $destinationKey are compared and only if they fit, the attribute group is added to the list. The $sourceKey is built using $mapProductAttributeGroupNamesSourceDest where we now look for Product Details instead of General.

This is actually the name, which is given by Magento by default. But as soon as a third party module adds a product attribute to the well known group 'General' from Magento 1, the group Product Details is renamed and so can not be found any more. To fix that, either rename the attribute group with the attribute_group_code = product-details back to Product Details before starting the migration or find the extension which causes the issue. For that set a breakpoint at \Magento\Eav\Setup\EavSetup::updateAttributeGroup with a filter for the correct id and check where the wrong setting comes from.

kanevbg commented 3 years ago

@norgeindian maybe this PR can fix your use case?

norgeindian commented 3 years ago

@kanevbg , that seems to be a good approach in my eyes, yes. Thanks for the hint. Would be great if #848 could be merged in general. In the meantime I will try it with it.

kanevbg commented 3 years ago

@norgeindian Please post back your experience with the PR.

norgeindian commented 3 years ago

@kanevbg , sorry, as Firegento fixed their issue with the wrong adding of the attribute group pretty fast, I did not try it with the PR. But can say for sure, that the migration of custom attributes in the "General" group work fine, when the groupname is not changed.

norgeindian commented 3 years ago

@PivitParkour94 , after fixing my issue with the General attribute group, I also ran into your issue with color and manufacturer when migrating from 1.9.4.5 to 2.4.0. I debugged a while and could actually find out why this happens.

The Leftover Step cleans all entries in catalog_product_entity_... tables where there are no corresponding entries in eav_entity_attribute (see \Migration\Step\PostProcessing\Model\AttributeSetLeftoverData::getLeftoverIds)
So the main question is, why there are no entries at all for 'color' and 'manufacturer' in table 'eav_entity_attribute'.
There would be two possibilities for having entries there.

They could either be handled as default destination attributes
They could be added as custom source attributes

Let's take a look why at the reasons why both cases don't work:

1. Handling as Default Destination Attributes

Destination attributes are added to each source attribute set in \Migration\Step\Eav\Data::createProductAttributeSetStructures,where they are taken from \Migration\Step\Eav\Model\Data::getDefaultProductEntityAttributes. This function takes all existing entries from eav_entity_attribute and adds them to the migrated attribute sets.
But 'color' and 'manufacturer' have no entry at all in this table, although they are added as default product attributes (see vendor/magento/module-catalog/Setup/CategorySetup.php)
The reason for that can be found in Magento\Eav\Setup\EavSetup::addAttribute, where attributes are only added to the table when a group is defined, or they don't have a value for the property user_defined
As our attributes are set to user_defined = true, they don't get added and this way are also not used in the migration process as default destination attributes.

Let's come to the second possibility:

2. Handling as Custom Source Attributes

This would happen in \Migration\Step\Eav\Data::migrateCustomEntityAttributes.
In \Migration\Step\Eav\Model\Data::getCustomAttributeIds all attributes from the source are compared to the already existent attributes in the destination table.
Unfortunately 'color' and 'manufacturer' are both already there and so get filtered out.
That is why these attributes don't get migrated as custom source attributes

Like I said in the beginning. After the EAV-Step there are no entries at all for these attributes in eav_entity_attribute and this is why all attribute values get removed in the PostProcessing Step. I tried to find out, if this issue is based on a change in the data migration tool or on a change in Magento itself, but was not successful. In general, I could not find any dependencies to third party modules, so I would assume, that these two attributes can currently not be migrated at all, which is a big bug in my eyes and should be fixed as soon as possible. If anyone has a proposal for a pull request, I would be happy to test and verify.

PivitParkour94 commented 3 years ago

@norgeindian We ended up just commenting out the \Migration\Step\PostProcessing\Model\AttributeSetLeftoverData step entirely because we were aware of all the data migrated and nothing was causing issues. It's a shame it didn't work out of the box. Otherwise, the migration tool has worked very well for a lot of migrations we have done

codebase-md commented 3 years ago

Today I followed @norgeindian with his fix for the firegento/firegento-magesetup2 (Fix attribute group) and checked the values for attribute group with attribute_group_code=product-details in the table eav_attribute_group. The attribute_group_name was initially 'General' as described. I changed it back to 'Product Details' and ran the migration again. Except for the attributes color and manufacturer (as mentioned above) all seems to be in order now for the attributes in the 'General' group of Magento 1. Thank you very much for the hint.

Ddcdom commented 3 years ago

@norgeindian your comment regarding the 'color' and 'manufacturer' attributes led me to a solution which I think is going to work in my case.

A fresh install of M2.3.5 comes both 'color' and 'manufacturer' product attributes, but of all the product attributes are created on install, these are the only two which are not assigned to the default attribute set.

As @norgeindian noted above in case 1, attributes which exist in M2, but which do not belong to an attribute set, are skipped and not imported.

My solution to this problem, is before the migration, go into the freshly installed M2 and add both the 'color' and 'manufacturer' attributes to the 'Default' attribute set (any group you want). Then do the migration and the 'color' and 'manufacturer' attributes are migrated as you'd expect.

I've no idea what would happen if you try migrating from an M1 store with multiple attributes sets, where the 'color' and 'manufacturer' attributes are assigned to more than one set. You might recreating the necessary attribute sets, but really I have no idea.

magento / data-migration-tool