Undefined Offset: in Migration/Step/Eav/Data.php

ChameleonDevil commented 4 years ago

Before I continue here:

For reference, I created this topic around the same time I added this issue last year: Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Edit 2021:

Please read the link above for the concern I raised:
Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed.... and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?

Original topic:

Preconditions

Magento 1.7.0.2 database
Magento 2.3.5 database, PHP 7.3

Steps to reproduce

Setup and install a clean new version of Magento 2.3.5
Install Migration Data tool v 2.3.5 composer require magento/data-migration-tool:2.3.5
Run setup:upgrade 4.1 Make sure that var/migration* is deleted so that you have a new migration from start. (in particular the file migration-tool-progress.lock). I would have assumed --reset does the same, but seems not.
Run bin/magento migrate:data path/to/config.xml that has configuration info from your source Magento 1 database and Magento 2 destination.

Expected result

A proper clean migration with no exceptions/errors/halts.

Actual result

Halts immediately on the following error

Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

Inside src/Migration/Step/Eav/Data.php on line 492: _It seems that it is attempting to update the record attribute_group_id with the populated entry inside mapAttributeGroupIdsSourceDest[id]_, but it is not available inside my data...

$record['attribute_group_id'] = $this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']];

attribute_group_id for $record is 4, but there seems to be a missing mapAttributeGroupIdsSourceDest[4] corresponding map source/destination

In fact there seems to be more missing indexes when I debugged

[Screenshot, logs]

_{[**2020-05-11 07:11:45] main.ERROR: Notice: Undefined offset: 4 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492}

0 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(494): Magento\Framework\App\ErrorHandler->handler(8, 'Undefined offse...', '/var/www/html/m...', 492, Array)**

1 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php(183): Migration\Step\Eav\Data->migrateCustomEntityAttributes()

2 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/AbstractMode.php(82): Migration\Step\Eav\Data->perform()

3 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(124): Migration\Mode\AbstractMode->runStage(Object(Migration\Step\Eav\Data), 'EAV Step', 'data migration')

4 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Mode/Data.php(69): Migration\Mode\Data->runData(Array, 'EAV Step')

5 /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Console/MigrateDataCommand.php(59): Migration\Mode\Data->run()

6 /var/www/html/mag_new/vendor/symfony/console/Command/Command.php(255): Migration\Console\MigrateDataCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

7 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(58): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

8 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(138): Migration\Console\MigrateDataCommand\Interceptor->___callParent('run', Array)

9 /var/www/html/mag_new/vendor/magento/framework/Interception/Interceptor.php(153): Migration\Console\MigrateDataCommand\Interceptor->Magento\Framework\Interception{closure}(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

10 /var/www/html/mag_new/generated/code/Migration/Console/MigrateDataCommand/Interceptor.php(104): Migration\Console\MigrateDataCommand\Interceptor->___callPlugins('run', Array, Array)

11 /var/www/html/mag_new/vendor/symfony/console/Application.php(1001): Migration\Console\MigrateDataCommand\Interceptor->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

12 /var/www/html/mag_new/vendor/symfony/console/Application.php(271): Symfony\Component\Console\Application->doRunCommand(Object(Migration\Console\MigrateDataCommand\Interceptor), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

13 /var/www/html/mag_new/vendor/magento/framework/Console/Cli.php(115): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

14 /var/www/html/mag_new/vendor/symfony/console/Application.php(147): Magento\Framework\Console\Cli->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

15 /var/www/html/mag_new/bin/magento(23): Symfony\Component\Console\Application->run()

16 {main} [] []

Additional notes

** This is a clean installation of v2.3.5 Magento as described above, with no pending migration-tool-progress.lock inside var/
My possible related query [ - regarding a much smaller Eav/Data.php file](https://github.com/magento/data-migration-tool/issues/812)

Questions

Is the map code from the last few commits that should append array mapAttributeGroupIdsSourceDest with entries not missing some processing that used to be there? (See my related query above) (I have done previous migrations pre v.2.3 successfully using the same source Magento 1 database)
Is my issue that I need to map groups as well? Where do I do this?

KZNcode commented 4 years ago

Having a very similar issue.

[2020-05-11 14:37:40][INFO][mode: data][stage: data migration][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
In ErrorHandler.php line 61:

  Notice: Undefined offset: 18 in /home/public_html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

Destination: Clean installed Magento 2.3.5-p1; data-migration-tool 2.3.5; Source: Magento 1.9.4.5

Previous magento versions import worked fine:

Destination: Clean installed Magento 2.3.4; data-migration-tool 2.3.4; Source: Magento 1.9.4.4

ChameleonDevil commented 4 years ago

Having a very similar issue.
[2020-05-11 14:37:40][INFO][mode: data][stage: data migration][step: EAV Step]: started
100% [============================] Remaining Time: < 1 sec
In ErrorHandler.php line 61:

  Notice: Undefined offset: 18 in /home/public_html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492
Destination: Clean installed Magento 2.3.5-p1; data-migration-tool 2.3.5; Source: Magento 1.9.4.5

Previous magento versions import worked fine:

Destination: Clean installed Magento 2.3.4; data-migration-tool 2.3.4; Source: Magento 1.9.4.4

This is most definitely the same issue, line 492 as explained above the array does not contain an item with index 18.

The method createMapProductAttributeGroupIds() populates mapAttributeGroupIdsSourceDest array but skips your group id 18. For me it was ID 4, where group 4 was '4-Meta Information', but the actual source database group id 4 is 'General Information', so '4-General Information'. There seems to be never a 'General Information' item in my migration, therefor mapAttributeGroupIdsSourceDest never contains an ID 4 when code on line 492 processes.

Potential fix - but read notes below I have created a commit to check for non-existing key. There are two changes in migrateCustomEntityAttributes(). This essentially just makes sure to skip non-existing IDs. After applying this to my local installation and running migrate:data on a backed up new M2.3.5 database, all the other steps continued without issue - took about 40 minutes.

Please note: I do not know which custom attributes we are now skipping on import with that commit, thats why my query of #812 still remains. I absolutely have no idea yet, and if the maintainers of this project could just explain that issue in detail.

victor-v-rad commented 4 years ago

'Undefined offset' error was fixed in ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6 and merged into 2.3-develop branch

ChameleonDevil commented 4 years ago

Yeah I fixed this with a similar way on my branch. Thanks for update!

Since fixing this, I am now trying to figure out why my categories do not show,

Every single one of them returns error 404. Anyways topic for another day

yogeshkhasturi commented 4 years ago

@ChameleonDevil Yes same issue here. products data is empty and category mapping the is there but form frontend it says "NO PRODUCT"

Did you find any solution?

ChameleonDevil commented 4 years ago

@yogeshkhasturi

I do not know if you really have the same problem I had here, but in my description I commented this:

Potential fix - but read notes below Go to that section above and read about the solution. I also believe that @victor-v-rad added a similar commit for this issue. Everything related to this issue is in my top comment, with my commit and @victor-v-rad replying later on with official commit.

This definitely fixed undefined index issues on this step, I have ran several migration items afterwards. Other failures, but not related to this issue anymore.

yogeshkhasturi commented 4 years ago

@ChameleonDevil

This definitely fixed undefined index issues on this step, I have ran several migration items afterwards. Other failures, but not related to this issue anymore.

Even my migration was completed after ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6 this fix but something was affected with the attribute group in my database. the database available in the database but the relation of the product with attributes is not there, if you can help with debugging it would be appreciable.

Thanks,

ChameleonDevil commented 4 years ago

Hi @yogeshkhasturi

Sorry just saw your message now.

Glad to hear that it fixed your first issue (of many :P).

This eav_attribute_group issue that you mentioned as far as I remember only occurs when you try to run bin/magento migrate:data --reset again once a previous migration already migrated all the attributes. I seem to recall the migration tool confused with the IDs of the groups, so just restart from the beginning.

What I did for myself to test and restore migrations (I have alot of other issues still not working :( ) was sort of the following:

Install Magento 2.3.5 completely, setup modules etc, setup backend settings that you want. Once you have a working database before even migrating M1 data, make a backup of this M2 database state.... because you will most likely restart alot of times because of migration issues. So create a full m2_full_install_before_migration.sql backup.
Create the same for Magento 1 database - until you are at the point where you can migrate without failures, you can restore this as well. NOTE: Delta migration is useless until you have successful migrate:data.
DROP M2 migrated database and restore from SQL in step 1.
VERY IMPORTANT: *Delete the var/migration ** entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.
So a complete DROP/RECREATE, RESTORE M2 DB, setup:upgrade, migrate data from M1 copy (locally) takes around an hour in my case because I have done it so many times.
I would ignore migrate:delta for now if I were you, you will have to restart so many times. Get data working, make notes of issues that are specific to your instance, so that you could do things step by step. My current list before even getting to the point where I am able to run migrate:delta is roughly 15 things that I have to change manually in the database beforehand!

Good luck!

yogeshkhasturi commented 4 years ago

@ChameleonDevil Thank you for the guidance this step was really new and seems to be helpful.

*VERY IMPORTANT: Delete the var/migration entries in Magento 2 folder. These files are used by the tool it seems to keep track of where you stopped previously. So since the migration does not work 100% yet, just restart from the beginning. (this might also be why you have group issues). I always delete these files.

estebansmith-ris commented 4 years ago

@ChameleonDevil I'm having the same issue with the 404 did you fix this issue?

ChameleonDevil commented 4 years ago

@ChameleonDevil I'm having the same issue with the 404 did you fix this issue?

@estebansmith-ris , Are you referring to categories that I had issues with error 404? Then yes, but since you are in this Eav/Data.php issue thread, Im not sure what you are talking about - there are plenty of 404 issues in M2.

If it is related to Categories, I am almost certain this issue I created a solution for #815 I had was related to the 404s for categories as well. (It was about two weeks ago, and since then trying to solve so many extra issues, that I can't recall.)

jerry-shopcommerce commented 4 years ago

I had the same issue with a migration from magento 1.9.4.5 to 2.3.5 today.

This comment gave me the idea to change the name of the default product attribute set to "Default" in the magento1 database: https://github.com/magento/data-migration-tool/issues/112#issuecomment-242317826

So far it seems to fix my problem.

RCady commented 4 years ago

I am having this issue in a migration from 1.9.3.2 to 2.3.5-p1. However, I have used the develop branches fix for the 'undefined offset' error and the migration completes. However, one of my custom product attribute sets has no attribute groups assigned to it and I'm thinking that this fix is part of the reason. Is it possible that ignoring these errors is causing my attribute groups to not be imported properly?

ChameleonDevil commented 4 years ago

When you do not have that fix applied, the data migration immediately fails, so that is not an option. At the point that fix is applied in Step/Eav/Data/migrateCustomEntityAttributes() the current implementation was created as follows by maintainer:

`if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) {
            continue;
        }`

The processing of the mapAttributeGroupIdsSourceDest beforehand in code seems to not have any group Ids mapped - no idea why.

There is this new updated commit 4 days ago (I assume this is the one you have) : #af40def29438fb that added another new condition:

if (empty($customAttributeIds)) { return; }

I seem to recall having this issue as well at one point where $customAttributeIds are somehow empty. See the history of the commits of Migration/Step/Eav/Data.php here : commits history.

The original issue discussed here was fixed in #ea9cc9f21d3634dd3695fd88f9c869d9535cb6a6.

Maybe you should revert one step commit back from the newest one and see if it changes anything?

My concern with this whole Step/Eav/Data.php file content was discussed here where I noticed some strange things: Why is the newest Eav Data file so much smaller.

Go read there, I do not even know if the current Data.php file we have are accurate - alot of processing code got removed in the commits I mention there. Also -- why is this there $this->progress->start(7); when previous older commits had $this->progress->start($this->getIterationsCount()); - hardcoded quantity vs dynamic.

Finally I just want to say good luck with Eav migration - it is a nightmare when you later discover all your missing attribute data. Hints: (Attribute sets vs attribute data), catalog_product_entity_int, catalog_product_super_link etc.

mhafizumer commented 3 years ago

'Undefined offset' error was fixed in ea9cc9f and merged into 2.3-develop branch

@victor-v-rad I have latest code,

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

But Still giving me error

Notice: Undefined offset: 19 in /var/www/html/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 286

What is the problem? even not found attribute_group_id is on `continue;

PascalBrouwers commented 3 years ago

I'm having the same issue, but then for category attributes. It seems that those attribute groups are never processed. Only products are being processed in createMapProductAttributeGroupIds().

hewgim commented 2 years ago

@victor-v-rad

We were also having the issue with:

Notice: Undefined offset: 7 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

We tried the fix mentioned above

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

but this resulted in loss of data. Product names, descriptions etc didn't get migrated over for one of our attribute sets.

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

Also if it helps anyone, we were migrating from 1.9.4.5 to 2.3.7-p1 but because there is no migration tool above 2.3.5 we used that instead and it seemed to work fine.

Hope that helps someone

Regards

Mark

ChameleonDevil commented 2 years ago

@victor-v-rad

We were also having the issue with:

Notice: Undefined offset: 7 in /var/www/html/mag_new/vendor/magento/data-migration-tool/src/Migration/Step/Eav/Data.php on line 492

We tried the fix mentioned above

if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

but this resulted in loss of data. Product names, descriptions etc didn't get migrated over for one of our attribute sets.

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

Also if it helps anyone, we were migrating from 1.9.4.5 to 2.3.7-p1 but because there is no migration tool above 2.3.5 we used that instead and it seemed to work fine.

Hope that helps someone

Regards

Mark

@hewgim

Can you perhaps please send us the full Migration/Step/Eav/Data.php that you managed to successfully migrate with.

Current commits (including the latest 2.4-develop) also uses if (!isset($this->mapAttributeGroupIdsSourceDest[$record['attribute_group_id']])) { continue; }

So, new migrations, even though it completes, does not really have entries in eav_entity_attribute that should attach eav_attribute_set with the different entities....

Before I continue, I am just putting this here for people skipping the top issue:

For reference, I created this topic around the same time I added this issue last year: Why is the newer 2.3.5 Step/Eav/Data.php so much smaller #812

Please read the link above for the concern I raised:
Summary: Looking at the current commits (2021) for Step/Eav/Data.php it still has less code (alot old code removed.... and hard-coded quantity $this->progress->start(7); vs OLDER commits: $this->progress->start($this->getIterationsCount()); ) as previous commits I mentioned in that topic; I wondered then, and still do: is this not the reason why Step/Eav/Data.php processing is giving us so much issues? Why were so much attribute processing removed, why are the dynamic getIterationsCount() not used anymore?

@hewgim, can you please assist us here with your last reply:

It would be great if you clear up this for us:

In the end we noticed that the attribute set with the missing data had originally been named "Default" and had been renamed to "Clothing". We renamed it back to "Default" and removed the code fix. When we reran the migration it completed error free. I suspect you have to have an attribute set named 'Default' as per the original Magento 1 installation.

"Default" as attribute set has 8x Default as name: (I am talking about the Magento 1 Source database):

EXPLAIN eav_attribute_set;

attribute_set_id
entity_type_id
attribute_set_name
sort_order

SELECT * FROM eav_attribute_set WHERE attribute_set_name = 'Default';

Each entity_type_id has its own Default attribute set, so there are 8x:

Results set:

1   1   Default 1
2   2   Default 1
3   3   Default 1
4   4   Default 1
5   5   Default 1
6   6   Default 1
7   7   Default 1
8   4   Default 1

When you mention '"Default" had been renamed to "Clothing"', do you mean your source Magento 1 database entry was renamed to Clothing or Magento 2 destination?

This is not clear from your explanation above,

If it's in the source database: then I think I am out of luck _since they are already called Default for all entity_typeid entries; I also don't know what to do, as we have over 120x different custom attribute sets (product attribute sets) in addition to those 8x built-in ones...

Would you perhaps please give us a workflow of what you did in the source & destination database to prepare (before running migration).

magento / data-migration-tool