QA: Automating the upgrade testing process for dotCMS using Docker

[bryanboza]

Task

There is a need to automate the upgrade testing process for dotCMS to streamline and ensure consistent testing when upgrading from one version to another. Currently, the upgrade process requires manual intervention, making it time-consuming and prone to errors. The goal is to automate the entire upgrade process using a script (in Bash, Python, or a similar language) that will manage the process in Docker. The user will provide the tag of the Docker image to migrate from and the tag of the image to migrate to, and the script should handle the rest of the process automatically. This will improve efficiency and accuracy when testing upgrades.

Requirements:

Automate the Upgrade Process: The script should automate the following steps:

• Pull the Docker image for the "from" version (starting version) based on a provided tag.
• Pull the Docker image for the "to" version (target version) based on a provided tag.
• Perform the necessary steps to upgrade the database and other relevant components during the migration.
• Run automated tests to verify the success of the upgrade.

Docker-Based Solution: The entire process should run in Docker, ensuring consistency across different environments.

Configurable Starter: The starter configuration should be configurable using a variable in the script.

Proposed Objective

Quality Assurance

Proposed Priority

Priority 2 - Important

Acceptance Criteria

• A script (written in Bash, Python, or a similar language) that automates the entire upgrade process in Docker.
• The script should accept two input parameters: the Docker image tag for the version to migrate from and the Docker image tag for the version to migrate to.
• The process should handle any necessary database migrations and verify the upgrade through automated tests.
• The script should be configurable, with a variable for setting the starter configuration (such as databases or other necessary components).
• Clear output logs should be generated to track the progress and results of the upgrade.
• Documentation about how to use

Assumptions & Initiation Needs

• Both the starting and target versions of the Docker images are available in a Docker registry.
• The script should be flexible enough to handle different versions of dotCMS with varying migration needs.

Quality Assurance Notes & Workarounds

• Test the script across different upgrade paths, especially between long-term support (LTS) versions and the latest trunk versions.
• Ensure that failures during the upgrade process are logged and handled gracefully.

Sub-Tasks & Estimates

• [ ] Explore the steps required to automate the upgrade process, including starting/stopping containers, applying migrations, and running automated tests post-upgrade.
• [ ] Determine the best approach for implementing the automation using Bash, Python, or another scripting language.
• [ ] Investigate how to configure Docker to use image tags provided by the user for both the starting and target versions.

[josemejias11]

QA Comment

Update on the coming task for the Upgrade process from LTS to Main. master_branch_upgrade.zip

[bryanboza]

@josemejias11, I've reviewed the provided script and here’s some feedback:

Step 1: We should allow the script to accept a specific version to upgrade_from. This adds flexibility for different versions.

Step 4: We need to add proper validations here. Simply printing the docker ps output isn’t enough. We should validate that the required containers are actually running.

Step 5: Since this is an automation script, it doesn't make sense to require manual intervention. The entire process should be automatic, and we should only receive a final result at the end.

Step 9: Before finishing, we need to run some assertions to ensure the process completed successfully. If it didn't, the script should return a clear error message.

[josemejias11]

QA Comment

Latest version worked on

Upgrade.zip

[bryanboza]

PR: https://github.com/dotCMS/core/pull/30480

[bryanboza]

@erickgonzalez, could you please provide me with a list of the versions we want to ensure compatibility between? I’d like to include one test case per version in this script and start running it in CI/CD

Example:

• Upgrade from last agile to main
• Upgrade from LTS xx to main

[erickgonzalez]

From a LTS POV:

• 23.01 LTS to main
• 23.10 LTS to main
• 24.04 LTS to main

From Agile:

• Last Agile to main

It will be good to test when the new LTS gets designated, but we can talk about that later:

• 23.01 LTS to new LTS
• 23.10 LTS to new LTS
• 24.04 LTS to new LTS