Getting 'Field discovery failed for Drupal core version 7.' when migrating

[geerlingguy]

Full message:

 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
  2/11 [=====>----------------------]  18% [error]  Attempt to create a field storage field_project_type with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  Attempt to create a field storage field_project_images with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  Attempt to create a field storage field_project_link with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  Attempt to create a field storage field_project_highlight with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  Attempt to create a field storage field_project_date with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
  3/11 [=======>--------------------]  27% [error]  Attempt to create a field storage field_tags with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  Attempt to create a field storage field_images with no type. (/var/www/html/web/core/modules/field/src/Entity/FieldStorageConfig.php:256) 
 [error]  The "comment" entity type does not exist. (/var/www/html/web/core/lib/Drupal/Core/Entity/EntityTypeManager.php:150) 

Maybe missing a field-related module? We'll look into it next week.

[geerlingguy]

Just wanted to drop this link here for future reference: Known issues when upgrading from Drupal 6 or 7 to Drupal 8

[geerlingguy]

I went ahead and added all the field modules that were seemingly missing.

Still getting this error:

 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field'

I also noticed that every time I run the command drush migrate-upgrade --legacy-db-key=drupal7 --legacy-root=https://www.jeffgeerling.com --configure-only it results in all the new migrations having an extra upgrade_ prepended to the machine name... so I would end up with migrations like upgrade_upgrade_upgrade_d7_thing.yml, and that's really annoying.

[geerlingguy]

Heh... so after all the images were migrated, I got:

 [warning] PDOStatement::execute(): MySQL server has gone away Statement.php:59
 [warning] PDOStatement::execute(): Error reading result set's header Statement.php:59
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_field_instance'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_entity_display'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_entity_form_display'

In Connection.php line 695:

  SQLSTATE[HY000]: General error: 2006 MySQL server has gone away: SELECT COUNT(*) AS expression  
  FROM                                                                                            
  (SELECT 1 AS expression                                                                         
  FROM                                                                                            
  {users} u                                                                                       
  WHERE u.uid > :db_condition_placeholder_0) subquery; Array                                      
  (                                                                                               
      [:db_condition_placeholder_0] => 0                                                          
  )                                                                                               

In Statement.php line 59:

  SQLSTATE[HY000]: General error: 2006 MySQL server has gone away 

So now I've found the limits for the weak MySQL DB in a container approach. I think I might have to switch to Drupal VM to get a bit more memory and speed behind the DB, or maybe just run the database off a separate docker container altogether.

[geerlingguy]

So for next episode, we'll kick off the migration again and see where that leads.

[geerlingguy]

It occurs twice:

...
 [notice] Processed 50 items (5 created, 0 updated, 0 failed, 45 ignored) - done with 'upgrade_d7_block'
 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 11 items (11 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field'
...

And the second time:

...
 [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_vocabulary'
 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 16 items (16 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_instance'
...

[geerlingguy]

I'm ditching both the colorbox and list_formatter modules on my D7 site for now; both were slightly less maintained than I'd like. Also, projects are kind of a step-child on my site, so I'm not going to put a lot of time into trying to make them fancy and/or perfect for this migration. (Eventually I might just merge them into blog posts and redirect...).

[geerlingguy]

That didn't help with these two particular issues. Still getting:

 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'
...
 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 15 items (15 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_instance'

[geerlingguy]

What's really annoying is there are no messages in any of the migrate_message_* fields, so it's really hard to debug what's going on here.

[geerlingguy]

This actually occurs four times in total. Grr.

[geerlingguy]

It actually appears six times in total. The rest of the migration is working great now, but these four notices make me nervous. It occurs during (before) the following migrations:

• upgrade_d7_field
• upgrade_d7_field_instance
• upgrade_d7_user
• upgrade_d7_comment
• upgrade_d7_field_formatter_settings
• upgrade_d7_field_instance_widget_settings

Related issue on Drupal.org (maybe): TypeError: Argument 1 passed to ConfigurableSearchPluginBase::setConfiguration() must be of the type array, null given

[geerlingguy]

I also read through Migration of Files from Dupal 7 to Drupal 8 with: [notice] Field discovery failed for Drupal core version 7., and checked the db tables like migrate_map_upgrade_d7_user for any rows with source_row_status <> 0, and there were none... so everything was migrated.

It looks like this error comes from the FieldDiscovery class. Might throw some debug message in there and re-run a migration to see what happens.

[geerlingguy]

Glancing at the FieldDiscovery class, I noticed that it logs a message (but doesn't log anything to the migrate_messages_* tables, so I looked in the watchdog log and found messages like:

Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance

Still not helpful yet.

[geerlingguy]

I wonder... looking at the migrate_plus.migration.upgrade_d7_field_instance.yml config file, I found that there's a mapping like:

comment_node_forum: comment_forum

Maybe I once had the Forum module enabled on the D6/D7 site and there's a lingering forum comment field somewhere that is trying to get migrated? Very strange.

[geerlingguy]

Somewhat related issue: Migration for forum and article comments: duplicate comment types and incorrect comment_entity_statistics.

[geerlingguy]

Hmm... but d7_comment_type.yml seems to indicate that's just a static mapping that doesn't really matter in the grand scheme of things. Will go back to debugging FieldDiscovery.

[geerlingguy]

Re-running any of the earlier-mentioned migrations triggers the message:

$ docker-compose exec drupal bash -c 'drush migrate-import --update upgrade_d7_user'
 [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: No database connection configured for source plugin d7_field_instance
 [notice] Processed 1 item (0 created, 1 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'

The $definition is:

Array
(
    [destination] => Array
        (
            [plugin] => null
        )

    [idMap] => Array
        (
            [plugin] => null
        )

    [source] => Array
        (
            [ignore_map] => 1
            [plugin] => d7_field_instance
        )

)

Using var_dump($source_plugin), I get that the plugin is a Drupal\field\Plugin\migrate\source\d7\FieldInstance. And then I checked var_dump($source_plugin->getIds()); and got:

array(3) {
  ["entity_type"]=>
  array(2) {
    ["type"]=>
    string(6) "string"
    ["alias"]=>
    string(3) "fci"
  }
  ["bundle"]=>
  array(2) {
    ["type"]=>
    string(6) "string"
    ["alias"]=>
    string(3) "fci"
  }
  ["field_name"]=>
  array(2) {
    ["type"]=>
    string(6) "string"
    ["alias"]=>
    string(3) "fci"
  }
}

And $source_plugin->getBaseId() gives d7_field_instance.

And if I check var_dump($source_plugin->__toString());, I get:

In SqlBase.php line 207:

  No database connection configured for source plugin d7_field_instance  

In Database.php line 361:

  The specified database connection is not defined: migrate  

[geerlingguy]

Hmm... so it looks like at least for the users migration, the problem is the first row in the users table in D7 is mostly-empty, uid 0 is the anonymous user. Not sure why that wouldn't be automatically ignored in the D7 upgrade:

[geerlingguy]

Well, that wasn't it, either. I deleted the uid 0 row and that made no difference. I'm stumped. I'll pick up the gauntlet again some other time.

[geerlingguy]

Hmm... mind might be about to be blown. Found this Twitter thread (https://twitter.com/DelishCreative/status/1222979764929146885) from @DelishCreative, and in the last message:

So the moral of the story kids: Don't name your migration DB group anything other than "migrate".

It might just be that I chose to set the key for my migration database to drupal7, and it should be migrate. If that's the case... WTH, Drupal!!!

$databases['drupal7']['default'] = [
  'database' => 'drupal',
  'username' => 'drupal',
  'password' => 'drupal',
  'prefix' => '',
  'host' => 'docker.for.mac.localhost',
  'port' => '3307',
  'namespace' => 'Drupal\\Core\\Database\\Driver\\mysql',
  'driver' => 'mysql',
];

[geerlingguy]

$ docker-compose exec drupal bash -c 'drush migrate-import --update upgrade_d7_user'
 [notice] Processed 1 item (0 created, 1 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'

SERIOUSLY. What. the...

So. Note to self: ALWAYS name your migrate db key migrate, until/unless whatever annoying bug that causes this issue is fixed. Maybe it has to do with a taxonomy-related migration? No clue, but I was going pretty deep into the rabbit hole tonight.

[geerlingguy]

And I twote: https://twitter.com/geerlingguy/status/1244834480642232322

[geerlingguy]

After above commit and reinstall/re-migrate:

[notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_type'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_dblog_settings'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_file_private'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_global_theme_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_settings'
 [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_image_styles'
 [notice] Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_menu'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_authorize'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_cron'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_date'
 [notice] Processed 1 item (0 created, 0 updated, 0 failed, 1 ignored) - done with 'upgrade_d7_system_file'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_mail'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_system_performance'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_theme_settings'
 [notice] Processed 4914 items (4914 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_url_alias'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_flood'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_mail'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_file_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_image'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_image_gd'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_logging'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_maintenance'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_rss'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_system_site'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_taxonomy_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_text_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_update_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_body_field'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_entity_display'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_block_content_entity_form_display'
 [notice] Processed 5 items (5 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_filter_format'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_custom_block'
 [notice] Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user_role'
 [notice] Processed 93 items (2 created, 0 updated, 0 failed, 91 ignored) - done with 'upgrade_d7_block'
 [notice] Processed 10 items (10 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_type'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_type'
 [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_vocabulary'
 [notice] Processed 15 items (15 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_instance'
 [notice] Processed 2674 items (2674 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_file'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_field'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_field_instance'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_entity_display'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_user_picture_entity_form_display'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_user'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_field'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_field_instance'
 [notice] Processed 2869 items (2869 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_blog_post'
 [notice] Processed 2 items (2 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_page'
 [notice] Processed 24 items (24 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_project'
 [notice] Processed 5853 items (5853 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_entity_display'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_entity_form_display'
 [notice] Processed 3 items (3 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_comment_entity_form_display_subject'
 [notice] Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_view_modes'
 [notice] Processed 28 items (28 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_formatter_settings'
 [notice] Processed 15 items (15 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_field_instance_widget_settings'
 [notice] Processed 1 item (1 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_revision_blog_post'
 [notice] Processed 4 items (4 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_revision_page'
 [notice] Processed 0 items (0 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_node_revision_project'
 [notice] Processed 3 items (0 created, 0 updated, 0 failed, 3 ignored) - done with 'upgrade_d7_node_title_label'
 [notice] Processed 1893 items (1893 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_tags'
 [notice] Processed 8 items (8 created, 0 updated, 0 failed, 0 ignored) - done with 'upgrade_d7_taxonomy_term_project_type'

Yay for no notices!

[gitressa]

Thanks for sharing this issue, as well as the solution @geerlingguy. I am glad to see you get no notices now. The upgrade to Drupal 8 using Drush documentation page has been updated with a Define the source database, to help others avoid running into this. Perhaps somebody can create an issue at some point?

[geerlingguy]

@gitressa - Good suggestion! I've opened 'Field discovery failed for Drupal core version 7' errors if migration source database key is not 'migrate'.

[gitressa]

Thanks @geerlingguy! Let's hope this gets fixed at some point. I can only imagine how many people have looked at migrating from Drupal 7 to Drupal 8, and given up due to its complexity. So I fear there will be a big reduction in the user base when Drupal 7 will reach EOL in November 2021.

Removing as many bumps as possible in the process is crucial, if Drupal is to have a future, and fixing this will definitely be a help.

[stephenplatz]

I have been searching for answers to this, went ahead and renamed my source\key to 'migrate' in my group shared configuration, but I'm still having the same issues with [notice] Field discovery failed for Drupal core version 7. Did this site have the CCK or Field module installed? Error: The module field is not enabled in the source site. My migration database key has always been migrate (not sure why the key in my migration group was different). I'm not even migrating from D7, my migration db is D6. Anyone else reading this still not getting it fixed?

[geerlingguy]

@stephenplatz - Note that the notice could also point to a field where the module providing that field doesn't exist on the Drupal 8 site too. In my case the notice was kind of masking a weird bug. But in your case it could be that there's actual field data that's not able to migrate due to a field not having a proper destination or something like that.

[stephenplatz]

If this is an actual error, then why did field discovery fail for Drupal 7, when I'm migrating from Drupal 6 to Drupal 8? I guess I wish that if this were an actual notice, that it would be more clear as to why it's happening, if for some other reason than the one mentioned in it. I will say that migration process plugins are nearly impossible for me to debug with xdebug and breakpoints, because of this check using a D7 process plugin, happening over and over again before I can hit my breakpoint.

[geerlingguy]

@stephenplatz - Yeah... it's not fun debugging this particular issue, since it seems it could be related to a variety of actual problems.

[webmasterpf]

Hello, Facing the same issue migrating D6 to D8, since I do this on a D8.8+ drupal fresh install. I don't know what changed between D8.7 and D8.8 but my custom migrations doesn't work anymore, even if I added DB connection info in settings.local. I f someone can explain how to fix this ? As a workaround, I do a migration on a D8.7 then I update it to D8.9 as normal way..not the simpliest fix :/

[geerlingguy]

@webmasterpf - This is a repo for my personal website, I can't provide general Drupal support here. But I'd recommend following this Drupal.org issue and providing feedback there, on Drupal Slack, etc.

[webmasterpf]

ok thanks.