Integrity Check Failed - Will not ignore Fields

[ghost]

I am attempting to run the Magento Data Migration Tool but running into issues. The following gets output into the migration.log

[2016-08-15 10:47:48][INFO][mode: data][stage: integrity check][step: EAV Step]: started
[2016-08-15 10:47:48][INFO][mode: data][stage: integrity check][step: Customer Attributes Step]: started
[2016-08-15 10:47:48][ERROR]: Source fields are not mapped. Document: customer_entity. Fields: mp_cc_is_approved
[2016-08-15 10:47:48][INFO][mode: data][stage: integrity check][step: Map Step]: started
[2016-08-15 10:47:52][ERROR]: Source documents are not mapped: belvg_events_events,belvg_events_participation,clnews_category,clnews_category_store,clnews_comment,clnews_news,clnews_news_category,clnews_news_store,cms_page_permission,cms_page_tree,ebizmarts_abandonedcart_abtesting,ebizmarts_abandonedcart_popup,ebizmarts_autoresponder_backtostock,ebizmarts_autoresponder_backtostock_alert,ebizmarts_autoresponder_review,ebizmarts_autoresponder_unsubscribe,ebizmarts_autoresponder_visited,flexslider_category,flexslider_group,flexslider_page,flexslider_product,flexslider_slide,flexslider_store,magemonkey_api_debug,magemonkey_async_orders,magemonkey_async_subscribers,magemonkey_bulksync_export,magemonkey_bulksync_import,magemonkey_ecommerce360,magemonkey_last_order,magemonkey_mails_sent,news,shipping_matrixrate,smtppro_email_log
[2016-08-15 10:47:52][ERROR]: Destination documents are not mapped: hozmegamenu,plazathemes_blog_category,plazathemes_blog_category_store,plazathemes_blog_post,plazathemes_blog_post_category,plazathemes_blog_post_relatedpost,plazathemes_blog_post_relatedproduct,plazathemes_blog_post_store,pt_bannerslider,pt_brandslider,pt_testimonial
[2016-08-15 10:47:52][ERROR]: Source fields are not mapped. Document: customer_group. Fields: allowed_payment_methods
[2016-08-15 10:47:52][ERROR]: Source fields are not mapped. Document: newsletter_subscriber. Fields: subscriber_firstname,subscriber_lastname
[2016-08-15 10:47:52][ERROR]: Source fields are not mapped. Document: sales_flat_order. Fields: ebizmarts_abandonedcart_flag,ebizmarts_magemonkey_campaign_id
[2016-08-15 10:47:52][ERROR]: Source fields are not mapped. Document: sales_flat_quote. Fields: ebizmarts_abandonedcart_counter,ebizmarts_abandonedcart_flag,ebizmarts_abandonedcart_token
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: Url Rewrite Step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: Log Step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: Ratings Step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: ConfigurablePrices step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: OrderGrids Step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: Tier Price Step]: started
[2016-08-15 10:47:52][INFO][mode: data][stage: integrity check][step: SalesIncrement Step]: started
[2016-08-15 10:47:53][INFO][mode: data][stage: integrity check][step: PostProcessing Step]: started

I have attempted to add some of these to in ignore tags in the map.xml within the field_rules tag

           <ignore>
                <field>customer_group.allowed_payment_methods</field>
            </ignore>
            <ignore>
                <field>customer_entity.mp_cc_is_approved</field>
            </ignore>
            <ignore>
                <field>newsletter_subscriber.subscriber_firstname</field>
            </ignore>

However the error still persists. Looking at other similar issues on here I have double checked my config.xml file and it is pointing to the correct map.xml file, the filenames are the same and so is the location.

<source>
        <database host="localhost" name="SourceDatabase" user="User" password="Password"/>
    </source>
    <destination>
        <database host="localhost" name="DestinationDatabase" user="User" password="Password"/>
    </destination>
    <options>
        <map_file>etc/ce-to-ce/1.9.2.4/map.xml</map_file>
        <eav_map_file>etc/ce-to-ce/map-eav.xml</eav_map_file>
        <eav_document_groups_file>etc/ce-to-ce/eav-document-groups.xml</eav_document_groups_file>
        <eav_attribute_groups_file>etc/ce-to-ce/eav-attribute-groups.xml</eav_attribute_groups_file>
        <log_map_file>etc/ce-to-ce/map-log.xml</log_map_file>
        <log_document_groups_file>etc/ce-to-ce/log-document-groups.xml</log_document_groups_file>
        <settings_map_file>etc/ce-to-ce/settings.xml</settings_map_file>
        <customer_map_file>etc/ce-to-ce/map-customer.xml</customer_map_file>
        <customer_document_groups_file>etc/ce-to-ce/customer-document-groups.xml</customer_document_groups_file>
        <customer_attribute_groups_file>etc/ce-to-ce/customer-attribute-groups.xml</customer_attribute_groups_file>
        <delta_document_groups_file>etc/ce-to-ce/deltalog.xml</delta_document_groups_file>
        <order_grids_document_groups_file>etc/ce-to-ce/order-grids-document-groups.xml</order_grids_document_groups_file>
        <map_document_groups>etc/ce-to-ce/map-document-groups.xml</map_document_groups>
        <class_map>etc/ce-to-ce/class-map.xml</class_map>
        <!--
        In case bulk_size=0 it will be auto-detected for every document.
        -->
        <bulk_size>0</bulk_size>
        <!--
        Set direct_document_copy = 1 for better performance.
        NOTE: 'source' and 'destination' databases MUST be placed on the same MySQL instance
        and 'destination' user MUST be granted with 'SELECT' permissions on 'source' database
         -->
        <direct_document_copy>0</direct_document_copy>
        <source_prefix />
        <dest_prefix />
        <auto_resolve_urlrewrite_duplicates>1</auto_resolve_urlrewrite_duplicates>
        <log_file>migration.log</log_file>
        <progress_bar_format>%percent%% [%bar%] Remaining Time: %remaining%</progress_bar_format>
        <upgrade_customer_password_hash>1</upgrade_customer_password_hash>
        <edition_migrate>ce-to-ce</edition_migrate>
        <edition_number>1.9.2.4</edition_number>
        <crypt_key>[Omitted]</crypt_key>
    </options>

I have searched these issues and the suggestions have not resolved it, this doesn't appear to have been occurring for many others, is there something I am missing?

[MagePsycho]

Make sure you use -r option while re-executing the command.

[ghost]

Hi @MagePsycho Sorry I forgot to mention, I am running the command with the -r option, as i thought that might help, but the same result unfortunately.

[MagePsycho]

Try placing the ignore tags in map-eav.xml and map-sales.xml

[ghost]

Same result I'm afraid, doesn't seem to be picking up the map entries?

[JDavidVR]

I'm having same issue :(

but I added them to map.xml.dist and also I changed map path in config.xml

<ignore>
    <field>newsletter_subscriber.subscriber_firstname</field>
</ignore>
<ignore>
    <field>newsletter_subscriber.subscriber_lastname</field>
</ignore>
<ignore>
    <field>sales_flat_order.ebizmarts_abandonedcart_flag</field>
</ignore>
<ignore>
    <field>sales_flat_order.ebizmarts_magemonkey_campaign_id</field>
</ignore>
<ignore>
    <field>sales_flat_quote.ebizmarts_abandonedcart_counter</field>
</ignore>
<ignore>
    <field>sales_flat_quote.ebizmarts_abandonedcart_flag</field>
</ignore>
<ignore>
    <field>sales_flat_quote.ebizmarts_abandonedcart_token</field>
</ignore>
<ignore>
    <datatype>quote_payment.cc_exp_month</datatype>
</ignore>
<ignore>
    <datatype>weee_tax.state</datatype>
</ignore>
<ignore>
    <datatype>sendfriend_log.ip</datatype>
</ignore>
<ignore>
    <datatype>rating_option_vote.remote_ip_long</datatype>
</ignore>
<ignore>
    <datatype>oauth_consumer.callback_url</datatype>
</ignore>
<ignore>
    <datatype>oauth_consumer.rejected_callback_url</datatype>
</ignore>
<ignore>
    <datatype>oauth_token.callback_url</datatype>
</ignore>

[victor-v-rad]

make sure you have these ignores inside tag

[julioxx22]

I Solved my problem adding those ignores in the

<source>

tag, I was adding it in the <destination> tag.

So if it says

"Source fields are not mapped. Document:" You have to add those fields in the <source> tag => map.xml file

[victor-v-rad]

Seems the issue is resolved. Hence closing it

[vincentwinkel]

[SOLVED] Sorry it's my bad, I had to put this update in map-customer.xml.

---

Hi, I reopen this issue because I've the same problem. I ran a first time migrate:data and got those errors :

[ERROR]: Source fields are not mapped. Document: customer_entity. Fields: oppcw_migrated
[ERROR]: Source documents are not mapped: aitoc_aitpermissions_advancedrole,aitoc_aitpermissions_approvedcategories,aitoc_aitpermissions_approvedproducts,aitoc_aitpermissions_editor_attribute,aitoc_aitpermissions_editor_tab,aitoc_aitpermissions_editor_type,aitsys_news,aitsys_status,aw_blog,aw_blog_cat,aw_blog_cat_store,aw_blog_comment,aw_blog_post_cat,aw_blog_store,aw_blog_tags,colorimage,ewcartreminder_blacklist,ewcartreminder_history,ewcartreminder_reminder,ewcore_config_data,ewcore_message,ewcore_module_summary,ewcore_system_message,frmtyp,giftcard_template,giftvoucher,giftvoucher_credit,giftvoucher_credit_history,giftvoucher_customer_voucher,giftvoucher_history,giftvoucher_product,giftvoucher_template,logistique,magenotification,magenotification_extension_feedback,magenotification_extension_feedbackmessage,magenotification_license,magenotification_log,mw_onestepcheckout,oppcw_alias_data,oppcw_customer_context,oppcw_external_checkout_context,oppcw_storage,oppcw_transaction,outofstocknotification_info,paypalauth_customer,tk_barcode_codes,tk_lucky_draw,tk_oauth,tk_order_x_carrier,tk_prescription_email,tk_prescription_fix,tk_prescription_form,tk_prescription_form_files,tk_prescription_formlab,tk_prescription_x_order,tk_refer_a_friend,tk_refer_a_friend_email,tk_refer_a_friend_sns,tk_tracking_code,tk_tracking_code_test,tk_tracking_numbers,tk_wishlist,tk_wishlist_item,tk_wishlist_item_option,tk_wms,vw_alerts,vw_banners,vw_photolinks,vw_shops_displays_config,vw_shops_global,vw_shops_group_entities,vw_shops_labels_config,vw_shops_product_entities,vw_shops_qrcodes_config,vw_shops_shop_entities,vw_shops_statistic_entities,vw_suppliers_orders
[ERROR]: Destination documents are not mapped: alerts_post,alerts_post_store,banner,boutique,boutique_store,externallinks_blog,externallinks_blog_store,externallinks_press,externallinks_press_store,magefan_blog_category,magefan_blog_category_store,magefan_blog_post,magefan_blog_post_category,magefan_blog_post_relatedpost,magefan_blog_post_relatedproduct,magefan_blog_post_store,magestore_bannerslider_banner,magestore_bannerslider_report,magestore_bannerslider_slider,magestore_bannerslider_value,mlx_license,mst_core_urlrewrite,mst_rewards_customer_referral_link,mst_rewards_earning_rule,mst_rewards_earning_rule_customer_group,mst_rewards_earning_rule_product,mst_rewards_earning_rule_queue,mst_rewards_earning_rule_website,mst_rewards_notification_rule,mst_rewards_notification_rule_customer_group,mst_rewards_notification_rule_website,mst_rewards_purchase,mst_rewards_referral,mst_rewards_spending_rule,mst_rewards_spending_rule_customer_group,mst_rewards_spending_rule_website,mst_rewards_transaction,testimonials_post,testimonials_post_store
[ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo. Fields: base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount,giftcard_refund_amount,giftvoucher_base_hidden_tax_amount,giftvoucher_hidden_tax_amount,giftcredit_base_hidden_tax_amount,giftcredit_hidden_tax_amount,oppcw_refund_id
[ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo_item. Fields: base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount,giftcard_refund_amount
[ERROR]: Source fields are not mapped. Document: sales_flat_invoice. Fields: base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount,giftvoucher_base_hidden_tax_amount,giftvoucher_hidden_tax_amount,giftcredit_base_hidden_tax_amount,giftcredit_hidden_tax_amount
[ERROR]: Source fields are not mapped. Document: sales_flat_invoice_item. Fields: base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount
[ERROR]: Source fields are not mapped. Document: sales_flat_order. Fields: base_giftvoucher_discount_for_shipping,giftvoucher_discount_for_shipping,base_giftcredit_discount_for_shipping,giftcredit_discount_for_shipping,base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount,giftvoucher_base_hidden_tax_amount,giftvoucher_hidden_tax_amount,giftcredit_base_hidden_tax_amount,giftcredit_hidden_tax_amount,giftvoucher_base_shipping_hidden_tax_amount,giftvoucher_shipping_hidden_tax_amount,giftcredit_base_shipping_hidden_tax_amount,giftcredit_shipping_hidden_tax_amount
[ERROR]: Source fields are not mapped. Document: sales_flat_order_item. Fields: base_gift_voucher_discount,gift_voucher_discount,base_use_gift_credit_amount,use_gift_credit_amount,giftvoucher_base_hidden_tax_amount,giftvoucher_hidden_tax_amount,giftcredit_base_hidden_tax_amount,giftcredit_hidden_tax_amount
[ERROR]: Destination fields are not mapped. Document: wishlist. Fields: cookie

In my map.xml, I added all source documents, source fields, destination documents and destination fields missing as following :

<map>
    <source>
        <document_rules>
            <ignore>
                <document>aitoc_aitpermissions_advancedrole</document>
            </ignore>
            <ignore>
                <document>aitoc_aitpermissions_approvedcategories</document>
            </ignore>
            <!-- etc -->
        </document_rules>
        <field_rules>
            <ignore>
                <field>customer_entity.oppcw_migrated</field>
            </ignore>
            <ignore>
                <field>sales_flat_creditmemo.base_gift_voucher_discount</field>
            </ignore>
            <!-- etc -->
        </field_rules>
    </source>
    <destination>
        <document_rules>
            <ignore>
                <document>alerts_post</document>
            </ignore>
            <ignore>
                <document>alerts_post_store</document>
            </ignore>
            <!-- etc -->
        </document_rules>
        <field_rules>
            <ignore>
                <field>wishlist.cookie</field>
            </ignore>
            <!-- etc -->
        </field_rules>
    </destination>
</map>

But after migrate:data -r I still got this error :

[ERROR]: Source fields are not mapped. Document: customer_entity. Fields: oppcw_migrated

Every other errors were resolved but not this one... Do you have an idea ? Thanks

---

[SOLVED] Sorry it's my bad, I had to put this update in map-customer.xml.

[liquidia]

Hi, Any one here who could tell if following fields are safe to ignore?

Source fields are not mapped. Document: catalog_product_option. Fields: validators Source fields are not mapped. Document: catalogrule. Fields: store_credit Source fields are not mapped. Document: catalogrule_product. Fields: store_credit Source fields are not mapped. Document: sales_flat_creditmemo. Fields: refundedstorecredit_amount,base_refundedstorecredit_amount,creditpoint_amount,base_creditpoint_amount Source fields are not mapped. Document: sales_flat_invoice. Fields: creditpoint_amount,base_creditpoint_amount Source fields are not mapped. Document: sales_flat_order_item. Fields: credits

As i am not sure if these are core magento data or 3rd party extensions data (document /fields)

Thanks

[pmoreno1980]

@liquidia i have the same question, why these fields are not mapping or ignore in the migration tool?

[2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_creditmemo. Fields: base_customer_balance_amount,customer_balance_amount,bs_customer_bal_total_refunded,customer_bal_total_refunded,base_gift_cards_amount,gift_cards_amount,gw_base_price,gw_price,gw_items_base_price,gw_items_price,gw_card_base_price,gw_card_price,gw_base_tax_amount,gw_tax_amount,gw_items_base_tax_amount,gw_items_tax_amount,gw_card_base_tax_amount,gw_card_tax_amount,base_reward_currency_amount,reward_currency_amount,reward_points_balance,reward_points_balance_refund [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_invoice. Fields: base_customer_balance_amount,customer_balance_amount,base_gift_cards_amount,gift_cards_amount,gw_base_price,gw_price,gw_items_base_price,gw_items_price,gw_card_base_price,gw_card_price,gw_base_tax_amount,gw_tax_amount,gw_items_base_tax_amount,gw_items_tax_amount,gw_card_base_tax_amount,gw_card_tax_amount,base_reward_currency_amount,reward_currency_amount,reward_points_balance [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_order. Fields: base_customer_balance_amount,customer_balance_amount,base_customer_balance_invoiced,customer_balance_invoiced,base_customer_balance_refunded,customer_balance_refunded,bs_customer_bal_total_refunded,customer_bal_total_refunded,gift_cards,base_gift_cards_amount,gift_cards_amount,base_gift_cards_invoiced,gift_cards_invoiced,base_gift_cards_refunded,gift_cards_refunded,gw_id,gw_allow_gift_receipt,gw_add_card,gw_base_price,gw_price,gw_items_base_price,gw_items_price,gw_card_base_price,gw_card_price,gw_base_tax_amount,gw_tax_amount,gw_items_base_tax_amount,gw_items_tax_amount,gw_card_base_tax_amount,gw_card_tax_amount,gw_base_price_invoiced,gw_price_invoiced,gw_items_base_price_invoiced,gw_items_price_invoiced,gw_card_base_price_invoiced,gw_card_price_invoiced,gw_base_tax_amount_invoiced,gw_tax_amount_invoiced,gw_items_base_tax_invoiced,gw_items_tax_invoiced,gw_card_base_tax_invoiced,gw_card_tax_invoiced,gw_base_price_refunded,gw_price_refunded,gw_items_base_price_refunded,gw_items_price_refunded,gw_card_base_price_refunded,gw_card_price_refunded,gw_base_tax_amount_refunded,gw_tax_amount_refunded,gw_items_base_tax_refunded,gw_items_tax_refunded,gw_card_base_tax_refunded,gw_card_tax_refunded,reward_points_balance,base_reward_currency_amount,reward_currency_amount,base_rwrd_crrncy_amt_invoiced,rwrd_currency_amount_invoiced,base_rwrd_crrncy_amnt_refnded,rwrd_crrncy_amnt_refunded,reward_points_balance_refund,reward_points_balance_refunded,reward_salesrule_points,onestepcheckout_order_comment,onestepcheckout_giftwrap_amount,ebizmarts_abandonedcart_flag,ebizmarts_magemonkey_campaign_id,rkl_web_source,rkl_utm_campaign,rkl_utm_source,rkl_utm_medium,rkl_utm_term,rkl_utm_content,cdr_order_comment [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_order_item. Fields: event_id,giftregistry_item_id,gw_id,gw_base_price,gw_price,gw_base_tax_amount,gw_tax_amount,gw_base_price_invoiced,gw_price_invoiced,gw_base_tax_amount_invoiced,gw_tax_amount_invoiced,gw_base_price_refunded,gw_price_refunded,gw_base_tax_amount_refunded,gw_tax_amount_refunded,qty_returned [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_quote. Fields: customer_balance_amount_used,base_customer_bal_amount_used,use_customer_balance,gift_cards,gift_cards_amount,base_gift_cards_amount,gift_cards_amount_used,base_gift_cards_amount_used,gw_id,gw_allow_gift_receipt,gw_add_card,gw_base_price,gw_price,gw_items_base_price,gw_items_price,gw_card_base_price,gw_card_price,gw_base_tax_amount,gw_tax_amount,gw_items_base_tax_amount,gw_items_tax_amount,gw_card_base_tax_amount,gw_card_tax_amount,use_reward_points,reward_points_balance,base_reward_currency_amount,reward_currency_amount,ebizmarts_abandonedcart_counter,ebizmarts_abandonedcart_flag,ebizmarts_abandonedcart_token [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_quote_address. Fields: base_customer_balance_amount,customer_balance_amount,gift_cards_amount,base_gift_cards_amount,gift_cards,used_gift_cards,giftregistry_item_id,gw_id,gw_allow_gift_receipt,gw_add_card,gw_base_price,gw_price,gw_items_base_price,gw_items_price,gw_card_base_price,gw_card_price,gw_base_tax_amount,gw_tax_amount,gw_items_base_tax_amount,gw_items_tax_amount,gw_card_base_tax_amount,gw_card_tax_amount,reward_points_balance,base_reward_currency_amount,reward_currency_amount [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_quote_address_item. Fields: gw_id,gw_base_price,gw_price,gw_base_tax_amount,gw_tax_amount [2017-07-19 22:14:43][ERROR]: Source fields are not mapped. Document: sales_flat_quote_item. Fields: event_id,giftregistry_item_id,gw_id,gw_base_price,gw_price,gw_base_tax_amount,gw_tax_amount

[lilu-box]

I had the same problems. Have to say, that the documentation could be more specific and detailed. The article http://oyenetwork.com/articles/data-migration-to-magento2-case-study/ and this thread have helped to risolve everything. Thank you all!

[pmoreno1980]

well the answer i have is that all those field are for magento enterprise, wird they said that im migrating EE to CE but not, im migrating CE to CE. weird.

[hirokapuriya]

Add the missing table in map.xml

        <ignore>
            <field>email_abandoned_cart</field>
        </ignore>
        <ignore>
            <field>temando_collection_point_search</field>
        </ignore>
        <ignore>
            <field>temando_order_collection_point</field>
        </ignore>
        <ignore>
            <field>temando_quote_collection_point</field>
        </ignore>

and Run

php bin/magento migrate:settings --reset /var/www/html/YOUR_ROOT_PATH/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.4/config.xml after not complete so run ----------------------------(auto) php bin/magento migrate:settings --auto /var/www/html/YOUR_ROOT_PATH/vendor/magento/data-migration-tool/etc/opensource-to-opensource/1.9.3.4/config.xml