[4.x]: 3 to 4 migration bug (I think?) with EntryTitleField not getting right namespace in table `fieldlayouttabs`

bossanova808 commented 1 year ago

What happened?

Description

Following my latest test migration from C3 to C4 (latest of both), I am seeing 34 of these for each page load:

2023-06-07 06:03:39 [web.WARNING] [craft\models\FieldLayoutTab::setElements] Invalid field layout element config: Invalid field layout element class: craft\fieldlayoutelements\EntryTitleField {"memory":11388768} 
2023-06-07 06:03:39 [web.ERROR] [yii\base\InvalidArgumentException] yii\base\InvalidArgumentException: Invalid field layout element class: craft\fieldlayoutelements\EntryTitleField in /var/www/html/vendor/craftcms/cms/src/services/Fields.php:1309
Stack trace:
#0 /var/www/html/vendor/craftcms/cms/src/models/FieldLayoutTab.php(288): craft\services\Fields->createLayoutElement(Array)
#1 /var/www/html/vendor/yiisoft/yii2/base/Component.php(180): craft\models\FieldLayoutTab->setElements(Array)
#2 /var/www/html/vendor/yiisoft/yii2/BaseYii.php(558): yii\base\Component->__set('elements', Array)
#3 /var/www/html/vendor/yiisoft/yii2/base/BaseObject.php(107): yii\BaseYii::configure(Object(craft\models\FieldLayoutTab), Array)
#4 /var/www/html/vendor/craftcms/cms/src/base/Model.php(78): yii\base\BaseObject->__construct(Array)
#5 /var/www/html/vendor/craftcms/cms/src/models/FieldLayoutTab.php(124): craft\base\Model->__construct(Array)
#6 /var/www/html/vendor/craftcms/cms/src/services/Fields.php(1237): craft\models\FieldLayoutTab->__construct(Array)
#7 /var/www/html/vendor/craftcms/cms/src/services/Fields.php(1215): craft\services\Fields->_createLayoutTabFromRow(Array, true)
#8 [internal function]: craft\services\Fields->craft\services\{closure}(Array)
#9 /var/www/html/vendor/craftcms/cms/src/services/Fields.php(1214): array_map(Object(Closure), Array)
#10 /var/www/html/vendor/craftcms/cms/src/services/Fields.php(1251): craft\services\Fields->getLayoutTabsById(Array)
#11 /var/www/html/vendor/craftcms/cms/src/services/Fields.php(1133): craft\services\Fields->_loadTabs(Array)
#12 /var/www/html/vendor/craftcms/cms/src/services/Sections.php(995): craft\services\Fields->getLayoutsByIds(Array)
#13 /var/www/html/vendor/craftcms/cms/src/services/Sections.php(972): craft\services\Sections->_entryTypes()
#14 /var/www/html/vendor/craftcms/cms/src/models/Section.php(326): craft\services\Sections->getEntryTypesBySectionId(3)
#15 /var/www/html/vendor/craftcms/cms/src/widgets/QuickPost.php(115): craft\models\Section->getEntryTypes()
#16 /var/www/html/vendor/craftcms/cms/src/controllers/DashboardController.php(77): craft\widgets\QuickPost->getSettingsHtml()
#17 /var/www/html/vendor/craftcms/cms/src/web/View.php(1619): craft\controllers\DashboardController->craft\controllers\{closure}()
#18 /var/www/html/vendor/craftcms/cms/src/controllers/DashboardController.php(76): craft\web\View->namespaceInputs(Object(Closure), '__NAMESPACE__')
#19 [internal function]: craft\controllers\DashboardController->actionIndex()
#20 /var/www/html/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#21 /var/www/html/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams(Array)
#22 /var/www/html/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction('index', Array)
#23 /var/www/html/vendor/craftcms/cms/src/web/Application.php(304): yii\base\Module->runAction('dashboard/index', Array)
#24 /var/www/html/vendor/yiisoft/yii2/web/Application.php(103): craft\web\Application->runAction('dashboard/index', Array)
#25 /var/www/html/vendor/craftcms/cms/src/web/Application.php(289): yii\web\Application->handleRequest(Object(craft\web\Request))
#26 /var/www/html/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest(Object(craft\web\Request))
#27 /var/www/html/web/index.php(12): yii\base\Application->run()
#28 {main} {"memory":11388984,"exception":"[object] (yii\\base\\InvalidArgumentException(code: 0): Invalid field layout element class: craft\\fieldlayoutelements\\EntryTitleField at /var/www/html/vendor/craftcms/cms/src/services/Fields.php:1309)"}

The fieldlayouttabs table seems to have data missing the full namespace:

If I re-save field layouts that have a Title field, then the number of errors per page drops for each one I save. I presume that means the fieldlayouttabs table is being updated. But obviously I'd prefer not to have to manually save 34 field layouts with every migration!

(conversation here: https://discord.com/channels/456442477667418113/456442477667418115/1115873349281779742)

Craft CMS version

4.4.13

PHP version

8.2

Operating system and version

DDEV

Database type and version

MySQL 8

Image driver and version

ImageMagick

Installed plugins and versions

bossanova808 commented 1 year ago

If I run this query, the errors stop:

UPDATE  craft_fieldlayouttabs
SET 
    elements = REPLACE(elements,
        'craft\\\\fieldlayoutelements\\\\EntryTitleField',
        'craft\\\\fieldlayoutelements\\\\entries\\\\EntryTitleField');

i-just commented 1 year ago

Hi, thanks for reaching out. I can only reproduce this behaviour under the following conditions:

set up Craft 3 site (3.8.13)
set up some sections with entry types that have titles
take a DB backup
run the update to Craft 4 (4.4.13) (update composer.json, run composer update, run php craft migrate/all)
check the fieldlayouttabs table - all is updated as expected
downgrade to Craft 3 again (update composer.json, run composer update)
import the DB backup from before the update (from step 3)
run the update to Craft 4 (4.4.13) again
check the fieldlayouttabs table - the settings column is NULL and elements still refers to the “old” namespace for the EntryTitleField

If, after step 7, I was to rebuild my project config based on what’s in the database (Utilities > Project Config > Rebuild the Config), the second update would have correctly updated the fieldlayouttabs.

In other words, you will encounter this issue if the project config files are already updated to Craft 4, but the database is for Craft 3.

Hope this helps!

bossanova808 commented 1 year ago

Well, sort of helps I suppose. But I find it a bit confusing.

So - if I am working on a big, complex Craft 3 -> Craft 4 upgrade, I need to regularly do test migrations of the latest production DB along the way.

(In the meantime, I am checking in PC etc as normal in the Craft 4 branch, and I never downgrade to Craft 3 in composer).

Of course, when I do those test migrations, I import my Craft 3 DB in my Craft 4 environment. After I have re-run the migration (i.e. both the Craft and Commerce upgrades) - on this 'fresh' DB from production, I rebuild the PC.

I thought this is what I am supposed to do - but it seems I am making a mistake? Am I supposed to nuke the PC before the upgrade or something?

Help would be appreciated!

bossanova808 commented 1 year ago

This is the process I actually follow:

[ ] Back up, then drop C4 db , then re-create empty db
[ ] Copy craft3-production* to \\wsl.localhost\Ubuntu\home\jeremy\ddev-projects\c4-dev\dbdump
[ ] Disable devMode on the local install for the upgrade - speeds it up a lot apparently!
[ ] cd4; ./dbdump/ddev-import-dbs.sh - this imports latest Craft 3 DB in DDEV - (10 mins)
[ ] ddev composer update (1 min)
[ ] ddev craft up (5 mins
[ ] ddev craft db/convert-charset (2 mins)
[ ] ddev craft commerce/upgrade - accept all the default field-names for the new custom fields (addressAttention etc) - takes ~ 1 hour if devMode is off, over 2 hours if in
[ ] ddev craft project-config/rebuild

i-just commented 1 year ago

Right, that’s a simpler and more “real” scenario, but the same thing will happen as your DB and project config are out of sync after the import. If you import your Craft 3 DB to your Craft 4 environment, you should also copy the Craft 3 config/project directory that matches that Craft 3 DB to the Craft 4 environment too.

bossanova808 commented 1 year ago

Right, I thought it was enough to just re-build it at the end from the migrated DB.

Thanks so much, I will try that in full tomorrow (hopefully my last test migration before I do the actual upgrade on Sunday, all going to plan).

i-just commented 1 year ago

Not a problem at all :)

brandonkelly commented 1 year ago

Came here to say the same thing :)

All migrations that make project config changes will start by making sure that the “incoming” project config YAML hasn’t already had the same migration applied. Otherwise the additional set of changes would end up overwriting the incoming changes. (Explained in more detail here.)

The migration that renames EntryTitleField is no exception:

https://github.com/craftcms/cms/blob/82629eef9b40a923471ae681ef8896865b3ee1bf/src/migrations/m220309_152006_rename_field_layout_elements.php#L26-L29

So if you ever need to re-run an update on dev, it’s best to discard your config/project/ folder as a starting point, so migrations don’t see that the YAML files have newer schema versions, indicating that the project config data shouldn’t be touched.

craftcms / cms