search

acquia / blt

Acquia's toolset for automating Drupal 8 and 9 development, testing, and deployment.
https://docs.acquia.com/blt/
GNU General Public License v2.0
442 stars 394 forks source link

drupal:sync:default:site and ckeditor 4 removal and other updb issues #4687

Closed loopy3025 closed 1 year ago

loopy3025 commented 1 year ago

I want to... Sync a database from production during Acquia Pipelines CI without any errors when a the a module has been uninstalled locally via composer.

It's not working because... We are in the process of removing drupal/ckeditor from our sites. When Pipelines runs blt drupal:sync:default:site, it does pull the database from production, but when it runs drupal:update, it throws "This module is installed on your site but is no longer provided..." type errors. This isn't a surprise because ckeditor is enabled on production, but we have composer removed it on our working branch being processed by the Pipelines CI.

 [error]   (Currently using Removed core module You must add the following contributed module and reload this page.
 * CKEditor [1]

This module is installed on your site but is no longer provided by Core.
For more information read the documentation on deprecated modules. [2]

[1] https://www.drupal.org/project/ckeditor
[2] https://www.drupal.org/node/3223395#s-recommendations-for-deprecated-modules
) [6.09 sec, 92.31 MB]

Then

>  [notice] Update started: system_post_update_enable_password_compatibility

>  [error]  The module ckeditor does not exist. 

>  [error]  Update failed: system_post_update_enable_password_compatibility

There is a similar issue with the in-place editing which we noticed on a different site. We had a role with this permission somehow after our d10 upgrade despite this is no longer a valid permission. When the database is copied, the block_content_post_update_sort_permissions update errors out because it is trying to sort the permissions and the in-place editing permission is no longer allowed:

>  [notice] Update started: block_content_post_update_sort_permissions

>  [error]  Adding non-existent permissions to a role is not allowed. The incorrect permissions are "access in-place editing". 

>  [error]  Update failed: block_content_post_update_sort_permissions

This one is fairly easy to resolve specifically because I can just remove that permission from prod without anyone complaining by using admin/config/development/configuration/single/import (because the drush rmp doesn't work for invalid permisisons either and it doesn't show up in the permissions list for me to un-check). However, I can see this being a real issue in other use-cases.

I feel like there's a chicken and egg issue here where the config needs to import in order to stop the errors from occuring, but then some db updates might not run or config import will fail because updates haven't been run on some modules yet.

Is there any way around this issue without having to leave the module installed in the codebase?

Otherwise, I believe the only way to run the sync is to leave drupal/ckeditor installed via composer during the removal process, then catch composer remove in a later release.

Detailed error output

Error output [error] (Currently using Removed core module You must add the following contributed module and reload this page. * CKEditor [1] This module is installed on your site but is no longer provided by Core. For more information read the documentation on deprecated modules. [2] [1] https://www.drupal.org/project/ckeditor [2] https://www.drupal.org/node/3223395#s-recommendations-for-deprecated-modules ) [5.38 sec, 92.31 MB] [info] Executing: /var/www/vendor/bin/drush updatedb:status --verbose --ansi --no-interaction --strict=0 --uri=*********--root=/var/www/docroot [5.4 sec, 92.76 MB] ------------------ ------------------------------- --------------- ------------------------------------------------------------------------------------------------------------------------------ Module Update ID Type Description ------------------ ------------------------------- --------------- ------------------------------------------------------------------------------------------------------------------------------ system 10100 hook_update_n 10100 - Remove the year 2038 date limitation. block_content 10100 hook_update_n 10100 - Update entity definition to handle revision routes. block_content 10200 hook_update_n 10200 - Remove the unique values constraint from block content info fields. comment 10100 hook_update_n 10100 - Remove the year 2038 date limitation. history 10100 hook_update_n 10100 - Remove the year 2038 date limitation. migrate 10100 hook_update_n 10100 - Remove the year 2038 date limitation. search_api_solr 8417 hook_update_n 8417 - Adds missing solr.FlattenGraphFilterFactory to index analyzers. smart_trim 10201 hook_update_n 10201 - Update Smart Trim more settings. Iterate through entity view displays and for any with Smart Trim as formatter type, move top level more link settings into more array. block_content block_library_view_permission post-update Update block_content 'block library' view permission. block_content move_custom_block_library post-update Moves the custom block library to Content. block_content sort_permissions post-update Update permissions for users with "administer blocks" permission. editor image_lazy_load post-update Enable filter_image_lazy_load if editor_file_reference is enabled. feeds add_feeds_feed_clear_action post-update Adds action plugin 'feeds_feed_clear_action'. feeds add_feeds_feed_import_action post-update Adds action plugin 'feeds_feed_import_action'. file add_permissions_to_roles post-update Grant all non-anonymous roles the 'delete own files' permission. media oembed_loading_attribute post-update Add the oEmbed loading attribute setting to field formatter instances. metatag remove_robots_noydir_noodp post-update Remove 'noydir', 'noodp' ROBOTS options from meta tag entity fields. responsive_image image_loading_attribute post-update Add the image loading settings to responsive image field formatter instances. responsive_image order_multiplier_numerically post-update Re-order mappings by breakpoint ID and descending numeric multiplier order. system enable_password_compatibility post-update Enable the password compatibility module. system linkset_settings post-update Add new menu linkset endpoint setting. system timestamp_formatter post-update Update timestamp formatter settings for entity view displays. text allowed_formats post-update Add allowed_formats setting to existing text fields. views boolean_custom_titles post-update Update Views config schema to make boolean custom titles translatable. views fix_revision_id_part post-update Fix '-revision_id' replacement token syntax. views oembed_eager_load post-update Add eager load option to all oembed type field configurations. views responsive_image_lazy_load post-update Add lazy load options to all responsive image type field configurations. views timestamp_formatter post-update Update timestamp formatter settings for views. ------------------ ------------------------------- --------------- ------------------------------------------------------------------------------------------------------------------------------ [notice] Module ckeditor has an entry in the system.schema key/value storage, but is missing from your site. More information about this error. [9.49 sec, 92.96 MB] [notice] Module entity has an entry in the system.schema key/value storage, but is not installed. More information about this error. [9.49 sec, 92.96 MB] [info] Executing: /var/www/vendor/bin/drush updatedb:batch-process 25202 --uri=*********** --root=/var/www/docroot [9.64 sec, 93.12 MB] > [notice] Update started: block_content_update_10100 > [notice] Added revision routes to Content block entity type. > [notice] Update completed: block_content_update_10100 > [notice] Update started: system_update_10100 > [notice] Update completed: system_update_10100 > [notice] Update started: block_content_update_10200 > [notice] Update completed: block_content_update_10200 > [notice] Update started: comment_update_10100 > [notice] Update completed: comment_update_10100 > [notice] Update started: history_update_10100 > [notice] Update completed: history_update_10100 > [notice] Update started: migrate_update_10100 > [notice] Update completed: migrate_update_10100 > [notice] Update started: search_api_solr_update_8417 > [notice] Update completed: search_api_solr_update_8417 > [notice] Update started: smart_trim_update_10201 > [notice] Update completed: smart_trim_update_10201 > [notice] Update started: block_content_post_update_block_library_view_permission > [notice] Update completed: block_content_post_update_block_library_view_permission > [notice] Update started: block_content_post_update_move_custom_block_library > [notice] Update completed: block_content_post_update_move_custom_block_library > [notice] Update started: block_content_post_update_sort_permissions > [notice] Update completed: block_content_post_update_sort_permissions > [notice] Update started: editor_post_update_image_lazy_load > [notice] Update completed: editor_post_update_image_lazy_load > [notice] Update started: feeds_post_update_add_feeds_feed_clear_action > [notice] Update completed: feeds_post_update_add_feeds_feed_clear_action > [notice] Update started: feeds_post_update_add_feeds_feed_import_action > [notice] Update completed: feeds_post_update_add_feeds_feed_import_action > [notice] Update started: file_post_update_add_permissions_to_roles > [notice] Update completed: file_post_update_add_permissions_to_roles > [notice] Update started: media_post_update_oembed_loading_attribute > [notice] Update completed: media_post_update_oembed_loading_attribute > [notice] Update started: metatag_post_update_remove_robots_noydir_noodp > [notice] There were no overridden Metatag records. > [notice] Update completed: metatag_post_update_remove_robots_noydir_noodp > [notice] Update started: responsive_image_post_update_image_loading_attribute > [notice] Update completed: responsive_image_post_update_image_loading_attribute > [notice] Update started: responsive_image_post_update_order_multiplier_numerically > [notice] Update completed: responsive_image_post_update_order_multiplier_numerically > [notice] Update started: system_post_update_enable_password_compatibility > [error] The module ckeditor does not exist. > [error] Update failed: system_post_update_enable_password_compatibility > [warning] Message: A new Solr field type has been installed due to configuration changes. It is > advisable to download and deploy an updated config.zip to your Solr server. > > [warning] Invalid placeholder (severity_level) with string: "%type: @message in %function (line %line of %file)." FormattableMarkup.php:245 > [warning] Invalid placeholder (exception) with string: "%type: @message in %function (line %line of %file)." FormattableMarkup.php:245 [error] Update aborted by: system_post_update_enable_password_compatibility [24.4 sec, 93.13 MB] [error] Finished performing updates. [24.4 sec, 93.07 MB] [Acquia\Blt\Robo\Tasks\DrushTask] Exit code 1 Time 24.83s [error] Failed to execute database updates! For troubleshooting guidance and support, see https://docs.acquia.com/blt/support/ The command failed. This often indicates a problem with your configuration. Review the command output above for more detailed errors, and consider re-running with verbose output for more information. [error] Command `drupal:update ` exited with code 1. For troubleshooting guidance and support, see https://docs.acquia.com/blt/support/

BLT doctor output

Output +----------------------+-----------------------------------------------+ | Property | Value | +----------------------+-----------------------------------------------+ | %paths.%root | /var/www/docroot | | %paths.%site | sites/default | | %paths.%modules | sites/all/modules | | %paths.%themes | sites/all/themes | | %paths.%config-sync | /var/www/config/sync | | %paths.%files | sites/default/files | | %paths.%temp | sites/default/tmp | | %paths.%private | /var/www/files-private/default | | admin-theme | claro | | alias-searchpaths.0 | /var/www/drush/sites | | blt-version | 13.7.1.0 | | bootstrap | Successful | | composer-version | Composer version 2.3.10 2022-07-13 15:48:23 | | config-sync | /var/www/docroot/../config/sync | | db-driver | mysql | | db-hostname | db | | db-name | default | | db-password | user | | db-port | 3306 | | db-status | Connected | | db-username | user | | drupal-settings-file | sites/default/settings.php | | drupal-version | 10.1.0 | | drush-alias-files.0 | /var/www/drush/sites/*****.site.yml | | drush-conf.0 | /var/www/vendor/drush/drush/drush.yml | | drush-conf.1 | /var/www/drush/drush.yml | | drush-script | /var/www/vendor/bin/drush | | drush-temp | /tmp | | drush-version | 11.6.0 | | files | sites/default/files | | install-profile | standard | | modules | sites/all/modules | | php-bin | /usr/local/bin/php | | php-conf.1 | /usr/local/etc/php/php.ini | | php-os | Linux | | php-version | 8.1.8 | | private | /var/www/files-private/default | | root | /var/www/docroot | | site | sites/default | | temp | sites/default/tmp | | theme | nyrc | | themes | sites/all/themes | | uri | http://*****.docksal.site | +----------------------+-----------------------------------------------+ +--------------------------------------+--------------------------------------------------------------+ | Check | Problem | +--------------------------------------+--------------------------------------------------------------+ | ConfigCheck:checkGitConfig | Git repositories are not defined in blt.yml. | | | Add values for git.remotes to blt.yml to enabled automated | | | deployment. | | NodeCheck:checkNodeVersionFileExists | Neither .nvmrc nor .node-version file found in repo root. | +--------------------------------------+--------------------------------------------------------------+

System information

mikemadison13 commented 1 year ago

My strong recommendation (due to situations like this) is to NOT sync databases during CI/CD. If you need config/content from your other sites, you can import those items specifically using config management and the default content module. There isn't a great work around for this. If the module is gone in the CI/CD container, it's going to throw this error.

loopy3025 commented 1 year ago

I'm starting to agree with you.

We're currently running pa11y tests during ci in order to catch accessibility issues. At the time we did this, it seemed to make sense to sync the database so we can test against existing content and full Views, all in context with how the users actually use the site. All of our testing went great and we've been happily using it for a couple of months now. Hindsight being 20-20, I'd have pushed harder to do a default content module of some kind to enable during CI instead of syncing the database. But now we're stuck with syncing on a LOT of sites if/until I can argue that methodology.

For the time being, I may meet halfway and switch the default site our stage environment. That way I can at least manipulate the database and re-run Pipelines as needed.

loopy3025 commented 1 year ago

I do really wish, though, that Drupal itself handled uninstalled modules better. It's pretty annoying that you need to have the module code in the codebase in order to uninstall a module properly. But I guess that's not really BLT's problem.