search

Islandora-Devops / isle-dc

ISLE 8 - Dockerized Islandora 8 Deployment orchestrated with docker-compose
MIT License
23 stars 60 forks source link

How to upgrade existing isle-dc site to Drupal 10 / Drush 12 / PHP 8.x (meta-issue) #356

Open ysuarez opened 1 year ago

ysuarez commented 1 year ago

In case it helps, I wanted to create this meta-issue to keep track of open issues and PRs that will help us run an existing ISLE-DC site with Drupal 10 / Drush 12 / PHP 8.x (meta-issue).

Apologies in advance if you do not find this meta-issue helpful, and I welcome your feedback.

Available isle-buildkit releases:

https://github.com/Islandora-Devops/isle-buildkit/releases https://github.com/Islandora-Devops/isle-buildkit/releases/tag/2.0.7

Related PRs:

Documentation PRs:

Related issues:

Related PRs for upgrade to PHP 8.2:

Related issues for upgrade to PHP 8.2:

@joshdentremont @aOelschlager @joecorall @nigelgbanks

aOelschlager commented 1 year ago

Adding some reference material. Solr upgrade docs: https://solr.apache.org/guide/8_0/major-changes /-in-solr-8.html#major-changes-in-solr-8 Matomo upgrade docs: https://matomo.org/faq/on-premise/update-matomo/

highermath commented 1 year ago

Is anyone working on this?

ysuarez commented 1 year ago

@highermath my understanding is that @joecorall has provided two key PRs (see above) that need testing and merging to push this upgrade forward. I think there might be one or more additional tasks needed for this issue to be closed, but merging those two PRs are key.

joshdentremont commented 1 year ago

Solr Search API requires updating the Solr config when you update the module to 4.3.0, which should be the same process as when we upgrade Solr from 7 to 8.

I am hoping to try updating Isle to 2.0.2 soon, but in the meantime, here are the instructions that worked for me to update the Solr config when I upgrade the Search API module.

Download config.zip from Drupal UI at /admin/config/search/search-api/server/default_solr_server Copy it into the solr container (docker compose cp ~/Downloads/solr_7.x_config.zip solr:/opt/solr/server/solr/ISLANDORA/conf) SSH into container (docker compose exec solr with-contenv bash) cd /opt/solr/server/solr/ISLANDORA/conf unzip solr_7.x_config.zip Replace all when prompted rm solr_7.x_config.zip exit docker compose restart solr make reindex-solr

noahwsmith commented 1 year ago

What happens if you run make solr-cores again? That seems very close to the steps that includes...

joecorall commented 1 year ago

@joshdentremont this works with TAG=main until we get a new 2.x tag with https://github.com/Islandora-Devops/isle-buildkit/pull/290

@noahwsmith can we get a bumped tag?

noahwsmith commented 1 year ago

yep! here you go https://github.com/Islandora-Devops/isle-buildkit/releases/tag/2.0.3

joshdentremont commented 1 year ago

@noahwsmith good catch. I just updated the solr search API to 4.3.0 and ran make solr-cores and it updated the config for me. That's much easier than the manual steps.

joshdentremont commented 1 year ago

I was able to update my site to Buildkit 2.0.3 and Drupal 10 and everything seems to be working except Matomo, which seems to be on version 3.13.5 and not be compatible with PHP 8.

I did this update on my local dev site. I have a staging and production site as well so I will comment again once I get it working on the staging site.

Below are two sets of steps. One that I think will work, and that we should test and possibly add to the documentation, and the exact steps I took that did work, but could be simplified.

One thing to note: this required removing all my data from Solr and recreating the core, then reindexing everything. It seemed like my existing core had been corrupted after the update to 8.x. There may be a way to avoid this, but I think you probably want to reindex anyway, because we are generating a new Solr config for 8.x.

Steps for documentation:

  1. git pull the latest isle-dc
  2. Edit the TAG variable in your .env file to 2.0.3 (or higher)
  3. make -B docker-compose.yml
  4. make pull && make down && make up
  5. docker compose exec -T solr with-contenv bash -lc 'rm -r server/solr/ISLANDORA'
    • Note that this deletes all data indexed in Solr
  6. make solr-cores
  7. make reindex-solr
  8. You should now have PHP 8 and Solr 8, but you will still be on Drupal 9
  9. Update your composer requirements for Drupal 10 and Drush 12
  10. docker compose exec -T drupal with-contenv bash -lc 'drush updb'

These are the exact steps I took that worked for me:

joshdentremont commented 1 year ago

On Staging/Production:

  1. git pull the latest isle-dc
  2. update .env to have TAG=2.0.3 (same as you have locally)
  3. make -B docker-compose.yml
  4. make pull && make down && make up
  5. docker compose exec -T drupal with-contenv bash -lc 'drush updb'
  6. docker compose exec -T solr with-contenv bash -lc 'rm -r server/solr/ISLANDORA'
  7. docker compose restart solr
  8. make solr-cores
  9. make reindex-solr

On both my dev site and staging site, Solr was acting up after the upgrade. On staging I had to remove the ISLANDORA directory, then restart Solr for it to clear out. Without doing that I was not able to run make reindex-solr

This also assumes you are using the Makefile to build, push, and pull your custom Drupal image. If you are not doing it that way, step 4 might be a little different, but the rest should be the same.

jefferya commented 1 year ago

Regarding Matomo, for the isle-buildkit TAG v1 to v2 upgrade, here's what locally, we found as the problem in the past.

isle-buildkit uses /var/www/matomo as a working directory to download and install Matomo during image creation (details). The Matomo version is v4 in isle-buildkit TAG v2

isle-dc creates a volume at mounted at /var/www/matomo (details).

    volumes:
      - matomo-config-data:/var/www/matomo

What is stored in the local storage volume overrides the Matomo files embedded in the isle-buildkit image TAG v2. Thus, if you start with TAG v1 and update to v2 isle-buildkit, you will get Matomo from your local volume (e.g., Matomo v3). However, if starting with a clean (i.e., empty volumes) then one would get the version of Matomo embedded in the isle-buildkit image. The documentation behind the behaviour.

Locally, we've moved away from Matomo before digging into a solution so I don't have solution. One thought, untested, update Matomo via the Matomo web interface which will update the contents of the storage volume as a first step (assumes there is a version of Matomo that works with both PHP 7 and 8). This might be off-base so use with caution.

g7morris commented 1 year ago

We have found changing the volume bind mount for matomo to 

    volumes:
    - matomo-config-data:/var/www/matomo/config:rw

Works well. One just needs to copy the config.ini.php file manually to upgrade.

joshdentremont commented 1 year ago

@aOelschlager and I found another issue where Cantaloupe was not working with Isle and buildkit 2.0.0 I just put in a PR for this https://github.com/Islandora-Devops/isle-dc/pull/360

The instructions are the same once this PR goes through. The first 2 steps of pulling isle-dc and running make -B docker-compose.yml will pull in those changes for Cantaloupe as well

joecorall commented 1 year ago

Hypercube running on islandora/hypercube:2.0.3 is not generating OCR for TIFF's https://github.com/Islandora/Crayfish/issues/176

Update: https://github.com/Islandora-Devops/isle-buildkit/pull/292 resolves the hypercube:2.x issue.

joshdentremont commented 1 year ago

Now that there is a fix for advanced search compatibility (https://github.com/Islandora/advanced_search/pull/41) I think everything is ready to go, except Matomo. I have created an issue for Matomo, at https://github.com/Islandora-Devops/isle-dc/issues/361

I also have a documentation PR being tested to give instructions on how to do the upgrade: https://github.com/Islandora/documentation/pull/2264

I am running Drupal 10 in staging now and planning to update production next week and everything seems to be working well.

I think we can probably close this once the documentation PR is merged, and the Matomo issue is dealt with. My only concern is that it seems most of us are not using Matomo, so I'm not sure how to address that issue. I think someone who has an existing site, that uses matomo, might need to do some testing of the options g7morris and jeffreya suggested above.

joshdentremont commented 1 year ago

Another issue that will affect anyone upgrading from 1.x to 2.x is mariadb errors in the logs. This doesn't seem to cause any major issues, but it does really clutter up the logs. I have a PR to fix this in Buildkit and have updated the instructions in the documentation PR above for how to deal with this.

See the discussion in slack: https://islandora.slack.com/archives/CM6F4C4VA/p1696525259478829

joshdentremont commented 1 year ago

The Mariadb issue is now fixed with buildkit 2.0.5. Upgrade instructions have been updated to include how to fix the database errors in the logs: https://islandora.github.io/documentation/installation/docker-maintain-isle/#version-1x-to-2x

joshdentremont commented 1 year ago

fcrepo imports are failing for me after switching to 2.x.

See https://github.com/Islandora-Devops/isle-buildkit/issues/304 for more details.

This is easily fixed by editing the Makefile. I have included an issue in isle-dc so we can discuss this at the next tech call. (https://github.com/Islandora-Devops/isle-dc/issues/368)