search

magento / data-migration-tool

Magento Data Migration Tool
https://devdocs.magento.com/guides/v2.3/migration/bk-migration-guide.html
Open Software License 3.0
334 stars 200 forks source link

Migrating data from 1.9.3.7 to 2.3.3 duplicates every product #804

Open JamesTheHacker opened 4 years ago

JamesTheHacker commented 4 years ago

I'm migrating the data from 1.9.3.7 to 2.3.3. The migration is successful. But, when I run: bin/magento index:reindex I get the following error:

Design Config Grid index has been rebuilt successfully in 00:00:00
Customer Grid index has been rebuilt successfully in 00:00:00
Category Products index has been rebuilt successfully in 00:00:01
Product Categories index has been rebuilt successfully in 00:00:00
Catalog Rule Product index has been rebuilt successfully in 00:00:00
Product EAV index has been rebuilt successfully in 00:00:01
Stock index has been rebuilt successfully in 00:00:00
Inventory index has been rebuilt successfully in 00:00:00
Catalog Product Rule index has been rebuilt successfully in 00:00:00
Product Price index has been rebuilt successfully in 00:00:01
Google Product Removal Feed index has been rebuilt successfully in 00:00:00
Google Product Feed index has been rebuilt successfully in 00:00:00
Catalog Search indexer process unknown error:
Item (Magento\CatalogInventory\Model\Stock\Status) with the same ID "2" already exists.

If I check the database with the following query: SELECT * FROM cataloginventory_stock_item WHERE item_id = 2; ... there is indeed 2 products with the same ID. Infact, there are duplicate products for every product in the database.

If I run the same query on the 1.9.3.7 database there is only 1 product. So, somewhere during the migration it's duplicated the products. Any idea what is happening here?

My config file for migration is below:

<?xml version="1.0" encoding="UTF-8"?>
<config xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
       xs:noNamespaceSchemaLocation="urn:magento:module:Magento_DataMigrationTool:etc/config.xsd">
   <steps mode="settings">
       <step title="Settings Step">
           <integrity>Migration\Step\Settings\Integrity</integrity>
           <data>Migration\Step\Settings\Data</data>
       </step>
       <step title="Stores Step">
           <integrity>Migration\Step\Stores\Integrity</integrity>
           <data>Migration\Step\Stores\Data</data>
           <volume>Migration\Step\Stores\Volume</volume>
       </step>
   </steps>
   <steps mode="data">
       <step title="Data Integrity Step">
           <integrity>Migration\Step\DataIntegrity\Integrity</integrity>
       </step>
       <step title="EAV Step">
           <integrity>Migration\Step\Eav\Integrity</integrity>
           <data>Migration\Step\Eav\Data</data>
           <volume>Migration\Step\Eav\Volume</volume>
       </step>
       <step title="Customer Attributes Step">
           <integrity>Migration\Step\Customer\Integrity</integrity>
           <data>Migration\Step\Customer\Data</data>
           <volume>Migration\Step\Customer\Volume</volume>
       </step>
       <step title="Map Step">
           <integrity>Migration\Step\Map\Integrity</integrity>
           <data>Migration\Step\Map\Data</data>
           <volume>Migration\Step\Map\Volume</volume>
       </step>
       <step title="Url Rewrite Step">
           <integrity>Migration\Step\UrlRewrite\Version191to2000</integrity>
           <data>Migration\Step\UrlRewrite\Version191to2000</data>
           <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
       </step>
       <step title="Log Step">
           <integrity>Migration\Step\Log\Integrity</integrity>
           <data>Migration\Step\Log\Data</data>
           <volume>Migration\Step\Log\Volume</volume>
       </step>
       <step title="Ratings Step">
           <integrity>Migration\Step\Ratings\Integrity</integrity>
           <data>Migration\Step\Ratings\Data</data>
           <volume>Migration\Step\Ratings\Volume</volume>
       </step>
       <step title="ConfigurablePrices step">
           <integrity>Migration\Step\ConfigurablePrices\Integrity</integrity>
           <data>Migration\Step\ConfigurablePrices\Data</data>
           <volume>Migration\Step\ConfigurablePrices\Volume</volume>
       </step>
       <step title="OrderGrids Step">
           <integrity>Migration\Step\OrderGrids\Integrity</integrity>
           <data>Migration\Step\OrderGrids\Data</data>
           <volume>Migration\Step\OrderGrids\Volume</volume>
       </step>
       <step title="Tier Price Step">
           <integrity>Migration\Step\TierPrice\Integrity</integrity>
           <data>Migration\Step\TierPrice\Data</data>
           <volume>Migration\Step\TierPrice\Volume</volume>
       </step>
       <step title="SalesIncrement Step">
           <integrity>Migration\Step\SalesIncrement\Integrity</integrity>
           <data>Migration\Step\SalesIncrement\Data</data>
           <volume>Migration\Step\SalesIncrement\Volume</volume>
       </step>
       <step title="Inventory Step">
           <integrity>Migration\Step\Inventory\Integrity</integrity>
           <data>Migration\Step\Inventory\Data</data>
           <volume>Migration\Step\Inventory\Volume</volume>
       </step>
       <step title="PostProcessing Step">
           <data>Migration\Step\PostProcessing\Data</data>
       </step>
   </steps>
   <steps mode="delta">
       <step title="Customer Attributes Step">
           <delta>Migration\Step\Customer\Delta</delta>
           <volume>Migration\Step\Customer\Volume</volume>
       </step>
       <step title="Map Step">
           <delta>Migration\Step\Map\Delta</delta>
           <volume>Migration\Step\Map\Volume</volume>
       </step>
       <step title="Log Step">
           <delta>Migration\Step\Log\Delta</delta>
           <volume>Migration\Step\Log\Volume</volume>
       </step>
       <step title="ConfigurablePrices step">
           <delta>Migration\Step\ConfigurablePrices\Delta</delta>
           <volume>Migration\Step\ConfigurablePrices\Volume</volume>
       </step>
       <step title="Tier Price Step">
           <delta>Migration\Step\TierPrice\Delta</delta>
           <volume>Migration\Step\TierPrice\Volume</volume>
       </step>
       <step title="Url Rewrite Step">
           <delta>Migration\Step\UrlRewrite\Version191to2000Delta</delta>
           <volume>Migration\Step\UrlRewrite\Version191to2000</volume>
       </step>
       <step title="OrderGrids Step">
           <delta>Migration\Step\OrderGrids\Delta</delta>
           <volume>Migration\Step\OrderGrids\Volume</volume>
       </step>
       <step title="SalesIncrement Step">
           <delta>Migration\Step\SalesIncrement\Delta</delta>
           <volume>Migration\Step\SalesIncrement\Volume</volume>
       </step>
       <step title="Inventory Step">
           <delta>Migration\Step\Inventory\Delta</delta>
           <volume>Migration\Step\Inventory\Volume</volume>
       </step>
   </steps>
   <source>
       <database host="localhost" name="sensational_toys" user="${db_username}" password="${db_password}" />
   </source>
   <destination>
       <database host="localhost" name="${db_name}" user="${db_username}" password="${db_password}" />
   </destination>
   <options>
       <map_file>etc/opensource-to-opensource/1.9.3.7/map.xml.dist</map_file>
       <eav_map_file>etc/opensource-to-opensource/map-eav.xml.dist</eav_map_file>
       <eav_document_groups_file>etc/opensource-to-opensource/eav-document-groups.xml.dist</eav_document_groups_file>
       <eav_attribute_groups_file>etc/opensource-to-opensource/eav-attribute-groups.xml.dist</eav_attribute_groups_file>
       <log_map_file>etc/opensource-to-opensource/map-log.xml.dist</log_map_file>
       <log_document_groups_file>etc/opensource-to-opensource/log-document-groups.xml.dist</log_document_groups_file>
       <settings_map_file>etc/opensource-to-opensource/settings.xml.dist</settings_map_file>
       <customer_map_file>etc/opensource-to-opensource/map-customer.xml.dist</customer_map_file>
       <customer_document_groups_file>etc/opensource-to-opensource/customer-document-groups.xml.dist</customer_document_groups_file>
       <customer_attribute_groups_file>etc/opensource-to-opensource/customer-attribute-groups.xml.dist</customer_attribute_groups_file>
       <delta_document_groups_file>etc/opensource-to-opensource/deltalog.xml.dist</delta_document_groups_file>
       <order_grids_document_groups_file>etc/opensource-to-opensource/order-grids-document-groups.xml.dist</order_grids_document_groups_file>
       <map_document_groups>etc/opensource-to-opensource/map-document-groups.xml.dist</map_document_groups>
       <class_map>etc/opensource-to-opensource/class-map.xml.dist</class_map>
       <tier_price_map_file>etc/opensource-to-opensource/1.9.3.7/map-tier-price.xml.dist</tier_price_map_file>
       <stores_map_file>etc/opensource-to-opensource/map-stores.xml.dist</stores_map_file>
       <!--
       In case bulk_size=0 it will be auto-detected for every document.
       -->
       <bulk_size>0</bulk_size>
       <!--
       Set direct_document_copy = 1 for better performance.
       NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
       and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
        -->
       <direct_document_copy>0</direct_document_copy>
       <source_prefix />
       <dest_prefix />
       <auto_resolve_urlrewrite_duplicates>0</auto_resolve_urlrewrite_duplicates>
       <log_file>migration.log</log_file>
       <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
       <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
       <edition_migrate>opensource-to-opensource</edition_migrate>
       <edition_number>1.9.3.7</edition_number>
       <init_statements_source>SET NAMES utf8;</init_statements_source>
       <init_statements_destination>SET NAMES utf8;</init_statements_destination>
       <crypt_key>${migration_crypt_key}</crypt_key>
   </options>
</config>

And I run like so: sudo -H -u "${user}" bash -c "${webroot}/bin/magento migrate:data ${migration_dir}/config.xml" ... which runs and completes successfully.

victor-v-rad commented 4 years ago

Hi @JamesTheHacker

Thank you for reporting this issue. cataloginventory_stock_item might have two products when cataloginventory_stock has more then one record. Usually it has only stock_id = 1 with stock_name = Default. But some extensions might add more if your webstore works with several stocks. If your store does not need more then one stork you can alter \Migration\Step\Inventory\Model\SourceItem::prepareSelect() to fetch only one record from cataloginventory_stock table and you should not have products duplicated in Magento 2 cataloginventory_stock_item table

JamesTheHacker commented 4 years ago

I think I understand. I don't need anything from the M1 database except for the products and categories. Is it possible to leave out any data from 3rd party plugins during the transfer? I'm not sure the best way to solve this.

iphigenie commented 4 years ago

I think I understand. I don't need anything from the M1 database except for the products and categories. Is it possible to leave out any data from 3rd party plugins during the transfer? I'm not sure the best way to solve this.

You can disable entire steps for each phase in your config.xml - inventory step is both in the data phase and the delta phase iirc