Flagship + IOs Drupal 10 Upgrade

[monicapatel21]

Support for Drupal 9 ends on November 1, 2023. Please review Flagship (and IO-only Modules) to estimate D10 upgrade. Assume Maria is dev, and leave Monica a multiplier guess just in case Jules, Katie, or someone else has to implement.

Timeline: As close to Nov 1 as possible. Gabe and Jaymz may also be updating devs.

Dev [8hrs]

This site's "Upgrade document"

https://docs.google.com/spreadsheets/d/1y3TWOjQ4Foa8YesOXpgSOVV9iZZizD38tAlCD3PIPj0/edit#gid=393797692

Maintenance page

-- place maintenance screenshot here --

Working Branch

d10-upgrade-prep - note, the ckeditor switch happened in this, but it is NOT ready. We need ckeditor_templates to be patched to work with it, and probably ckeditor_accoridions too. See https://github.com/wri/wriflagship/commit/f4db77278fe1d135ea8602b697742aafa979e5aa, especially the large difference in config/filter.format.full_html.yml

Estimate for phase 2

48

Update Branch

d10-upgrade-prep

Part 1: Prep

At the end of this work, the site be in a state closer to be ready for a Drupal 10 update. Some of its modules may have been updated. There will be a spreadsheet (the "Upgrade document") with accurate information about how much time the upgrade should take.

Step 1 - organization things:

• [x] Start the deployment "Upgrade document" by copying https://docs.google.com/spreadsheets/d/1yMIwE7-i-ydaUvN2qmZ9z7elLxNPxmBvgxpJ0rbZtaM/edit?usp=sharing into your site's drive folder and renaming it.
• [x] Add a link to your version of the "Upgrade document" to this ticket above.
• [x] Inventory the list of modules used on the live site. A quick way to get that is running: terminus drush YOURSITE.live -- pml --format=csv --no-core > YOURSITE-modules.csv. You can then upload that list to the google drive "Upgrade document" you made in the last step, and alter the columns as needed. Note: Sometimes this command times out on the live site. You can try running it without the > and just copy/paste the output into a csv file if this happens. Then upload the csv to the Upgrade Document with "Append to the current sheet"
• [x] Review the list of modules that are on the site but uninstalled. Could any be removed from the composer.json file? Indicate that these can be removed from composer.json in the "Upgrade document" by setting their "Version post-upgrade" value to "Removed from the site". Note: Many uninstalled modules are submodules of enabled modules. Only add the "Removed from the site" designation to modules in your composer.json file.
• [x] Review the list of modules that are enabled on the site. Are there any modules that are no longer in use (like migration modules)? Indicate that they can be uninstalled in the "Upgrade document" by setting their "Version post-upgrade" value to "Uninstalled, to be removed"

Step 2 - cleaning up the code:

• [x] Start a new branch for this work. Do a robo install to make sure everything's clean on your local.
• [x] Uninstall any modules in the "Upgrade document" where the status is "Uninstalled, to be removed". Add this module to the list of items to be uninstalled from composer in the "Followup" section of this ticket.
• [x] Remove composer remove xx/xx any modules in the "Upgrade document" where the status is "Removed from the site".
• [x] The site should already be running PHP 8.1. If it isn't, follow these instructions: https://library.thinkshoutlabs.com/articles/upgrade-drupal-site-php-8
• [x] Switch the environment you're working in to PHP 8.1. (sphp 8.1)
• [x] Get the site to the latest version of Drupal 9.
• [x] Review the CKEditor 4 -> 5 D.O. article.
• [x] Add CKEditor 5 before removing CKEditor 4. Make sure to switch any CKEditor text editors to CKEditor 5 using the toggle in the text editor admin screen.
• [x] Remove Bene media (see https://library.thinkshoutlabs.com/articles/how-uninstall-benemedia)
• [x] Make sure the thinkshout/robo-drupal requirement is at v4.0.0 or higher. composer require --dev "thinkshout/robo-drupal:^4.0.0"
• [x] Ensure the upgrade_status module is installed en enabled: composer require --dev drupal/upgrade_status && drush en upgrade_status.
• [x] Push up this branch and note it in this ticket below under "Working Branch".

Step 3 - finalizing the update list:

• [x] Reviewing the /admin/reports/upgrade-status page, note any modules in the "UPDATE" and "COLLABORATE WITH MAINTAINERS". Indicate the module status in the "Upgrade document".
  • [x] If a module/theme has a d10 version, set their "Version post-upgrade" value to "Needs upgrade"
  • [x] If a module/theme has no d10 version, set their "Version post-upgrade" value to "Needs patch"
  • [x] If a module/theme is custom code, set their "Version post-upgrade" value to "Issues found:"
• [x] Any modules in the "Upgrade document" that do not already have a status can be set to "No change". These modules won't need to be tested!

Step 4 - Estimating

• [x] Review your module list in the "Upgrade document". Your estimate for proceeding should be:
  • [x] 1 hour for each module whose status is "Needs patch"
  • [x] 30 minutes for each module whose status is "Needs upgrade"
  • [x] 1 hour for any Custom module/themes.
  • [x] 15 minutes PER ISSUE FOUND for any modules that have "Issues found: X". This is in addition to the 1 hour above, as these will likely all be custom modules. NOTE: some custom modules will have only one issue, which is changing the info.yml file to be compatible with Drupal 10 (see example). This issue will require minimal time, so you may be able to scale down this part of the estimate if this is the extent of custom module issues.
• [x] Adjust this number for the complexity of the site. Note: This formula is already in the spreadsheet.
• [x] Add time for updating Behat tests, if the site uses those, and VR tests, if the site uses those.
• [x] If the site doesn't already have VR tests, you can set up some simple ones using the instructions for How to set up backstop not on Pantheon Build Tools here. At the very least, make a note to add VR tests into the Followup items section of this ticket.
• [x] Evaluate the complexity of their CKEditor implementation and select the appropriate dropdown option on the Estimate tab . This is a big change from D9 to D10 and may require additional time.
• [x] Establish how comfortable the client will be to QA their site. On the Estimate tab, select the appropriate option from the dropdown.
• [x] If the site currently has Bene Media and will need it removed, toggle the Bene Media dropdown in the Estimate tab accordingly.
• [x] Add the estimate total from the Estimate tab this ticket

Note that if you know your custom code is complicated, you can up the amount of time as you see fit.

Part 2: Update

Dev [24-48hrs]

Deployment 2: Upgrade

Step 1 - Maintenance page approval

• [x] Put your local site in Maintenance mode. Take a screenshot of that page and attach it to this ticket under "Maintenance page" and ask the client to review/approve it. -- skipping for WRI since it hasn't changed.

Step 2 - Code updates

• Keep the Library article handy so you can look up any issues you encounter. Hopefully they have already been documented.
• [x] If it's not already enabled, enable the "upgrade_status" module.
• [x] Review Deprecated and obsolete article.
• [x] Switch admin to a supported theme such as Claro or [Gin](https://www.drupal.org/
• [x] Check on the advanced editor link in the WYSIWYG
• [x] Remove uninstalled or deprecated modules. In /admin/reports/upgrade-status, each module listed under "Remove" that is not enabled on live can be removed from composer using composer remove blah blah. Add any modules that ARE enabled on live (migration modules for example) to the list of follow-up tasks to remove. If there isn't a d10 equivalent to a contributed module OR if you have a custom module that you want to disable and delete but you don't want to deal with upgrading, you can strip it down to just the info file and rename it "ghost_of_modulename" and then you can delete this with the next deploy. This needs to be here to not have any errors on initial D10 install. See video for better explanation.
• [x] Add deprecated themes such as Classy as contrib classy, if you've used them as the base for the TS theme (we always do this).
• [x] In /admin/reports/upgrade-status, update each modules listed under "UPDATE" using something like composer update drupal/modulename --with-dependencies.
• [x] In /admin/reports/upgrade-status, look at the modules remaining under "UPDATE" and "COLLABORATE WITH MAINTAINERS" and see which modules need a major version upgrade. Handle those with something like composer require drupal/modulename -W.
• [x] In /admin/reports/upgrade-status, look at what's left under "UPDATE" and "COLLABORATE WITH MAINTAINERS" and check all issue pages for these on drupal.org to see if there's a D10 patch available. If any modules do not have a D10 update or patch, note them below under "List modules that do not have a D10 compatible version of patch available." These might be good to discuss with other devs to see how to best approach these.
• [x] Custom modules will likely show up in the FIX MANUALLY section. For these, fix any issues that upgrade_status has found and rescan until they are all resolved. Note any that do not currently have D10 versions or that can't be fixed and flag them in tech talk for assistance.
• [x] drush updb and drush cex
• [x] When all D10 blockers are resolved in /admin/reports/upgrade-status and drupal-check, do the Drupal 10 upgrade. Run this command composer require drupal/core-recommended. This should update you to the latest version of drupal core on the 10.x branch. Note: If you still run into conflicts, try removing everything in your dev dependencies before running the update command. Don't forget to add everything back once the upgrade is successful.
• [x] Make sure all behat and visual regression tests pass.
• [x] to add - Removal of Bene Media
• [x] IMPORTANT! Enable the Password Compatibility Module. If you do not all users will need to reset their password and they will be cranky.

Step 3 - Finalize testing instructions

• [x] Fill in the "Version post-upgrade" values for all modules that have been updated in the "Upgrade Document".
• [x] Document manual tests in the "Upgrade document". You should write at least one set of testing instructions for all modules that have just been altered in any way.
• [x] In your test instructions you might write a test that's something like "Go to /page and review it looks right". If this site has Visual Regression testing set up, add the url of that page to the VR tests. Let the robots test for you!
• [ ] Make sure all behat and visual regression tests pass.
• [x] Update config_split so upgrade_status module is enabled on dev only
• [x] When pushing this work to a multidev, note any unusual deployment steps in the section below.
• [ ] Fill in the Testing instructions section of this ticket.
• [ ] Dev: Clear out the drupal logs at /admin/reports/dblog/confirm so it'll be easy to see errors or warnings found when testing the dev environment
• [ ] Hand the ticket over to the client for testing.
• [ ] Review the drupal logs at /admin/reports/dblog after tests are completed and note any errors, notices, or warnings that might be related to the update.

Notes

Testing Instructions:

• [ ] Run through the list of tests for Module updates in the "Upgrade document" (link).
• [ ] Note any bugs that happen on the https://[multidevurl].pantheonsite.io/ site that DO NOT happen on the https://[live-url]/ site. (If it's a bug on the live site, make a new ticket for it!)
• [ ] Consider any parts of the site that are high-priority that might not have been checked as part of the QA list and add a new row for each, tagged with "Non-module" tab.
• [ ] Update each row with Pass/Fail. If QA for an item fails, create a ticket and link to it in the "Ticket, if failed" section.

Deployment steps

• [ ] Prepare the client -- this update will need to be done while the site is in maintenance mode. Work with the PM to schedule that.

• [ ] Deploy to test and pause for re-checking -- the client should look over the highest-priority QA items again on the test multidev (not develop) using a recent copy of the live site's database, a day before the go-live deploy. You should also run any VR tests comparing test and live at this time.

• [ ] If using Solr, clear the index. Go to the server. There is a link at the bottom to "delete all indexed data". Click the link, as shown here:

  

• [ ] Push to live.

  • [ ] upload the code,

  • [ ] run a drush cr,

  • [ ] then run drush updb until there are no more pending updates. It might take a few tries to get them all.

  • [ ] Then run a drush cim.

  • [ ] if upgrading to solr, click index now in config:

    

  • [ ] You might need to take the site out of maintenance mode when all is done.

• [ ] Make a new ticket containing all the Followup items noted below.

Followup items:

• [ ] Remove from composer: textfield_counter

This Ticket is Done When:

• [ ] The site has been updated to Drupal 10 and works properly.

When finished

• [x] Update ThinkShout internal spreadsheet and tag Monica

[mariacha]

Substeps todo for modules not listed as compatible:

• [x] Finish fixes on wri_sites.
• [x] Dynamic Entity Reference - set to 3.x
• [x] Config_suite, test out the Call to deprecated method stopPropagation() of class Symfony\Component\EventDispatcher\Event: since Symfony 4.3, use "Symfony\Contracts\EventDispatcher\Event" instead error on config_suite.
• [ ] Reroll https://www.drupal.org/project/drupal/issues/3106205 which has an update hook conflict with D10.

[shannon-paton]

Hi @mariacha , flagging an issue that I noticed on live. On staff profile pages, it seems that only up to one office is showing on staff profiles when not logged in. When logged in, it looks as expected.

Example: old version: https://strategy-wriflagship.pantheonsite.io/profile/clarissa-lehne new version: https://www.wri.org/profile/clarissa-lehne

[mariacha]

It looks like in that case the second office "United Kingdom" is unpublished on live. It redirects to Europe:

https://wri.org/europe/united-kingdom

I could change the display to use the "Label (with unpublished)" display setting. That works alright on develop with the link seeming to go to the right place (europe): https://develop-wriflagship.pantheonsite.io/profile/clarissa-lehne

but do we want to show that office if it's unpublished? Maybe it was unpublished on accident?

[shannon-paton]

Hi @mariacha ah right, that makes sense, not a bug! We combined our Europe and United Kingdom pages into one, but I think we can still show both offices. What you have on develop would work for now, thanks!

[mariacha]

Alright, great! Done: https://www.wri.org/profile/clarissa-lehne

[shannon-paton]

Looks good, thanks!

[mariacha]

Since we've got QA tickets for all the multidevs now, I'm going to close this issue in favor of those. But we can re-open if new flagship issues show up!