Trying to run different combinations of database or PHP in one run fails

[NeillM]

I built a job that looks like:

pipeline {
    agent {label 'testing'}

    options {
        checkoutToSubdirectory('files/converter/onedrive')
        disableConcurrentBuilds()
    }

    stages {
        stage("Plugin CI") {
            steps {
                withMoodlePluginCiContainer(php: '7.4', db: 'mysql') {
                    moodlePluginCiInstall("--branch MOODLE_311_STABLE --plugin ./files/converter/onedrive")

                    moodlePluginCi 'phplint'
                    moodlePluginCi 'phpcpd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpmd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'codechecker --max-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpdoc', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'validate', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'savepoints', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'mustache', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'grunt --max-lint-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpunit'
                }
                withMoodlePluginCiContainer(php: '8.1', db: 'postgres') {
                    moodlePluginCiInstall("--branch MOODLE_401_STABLE --plugin ./files/converter/onedrive")

                    moodlePluginCi 'phplint'
                    moodlePluginCi 'phpcpd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpmd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'codechecker --max-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpdoc', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'validate', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'savepoints', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'mustache', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'grunt --max-lint-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpunit'
                }
            }
        }
    }
}

It runs the first set of tests fine, however on the second moodle-plugin-ci install I get the following output:

moodle-plugin-ci install --db-type pgsql --db-user jenkins --db-pass jenkins --db-host 127.0.0.1 --branch MOODLE_401_STABLE --plugin ./files/converter/onedrive

In Validate.php line 29:

Failed to run realpath('./files/converter/onedrive')

install [--moodle MOODLE] [--data DATA] [--repo REPO] [--branch BRANCH] [--plugin PLUGIN] [--db-type DB-TYPE] [--db-user DB-USER] [--db-pass DB-PASS] [--db-name DB-NAME] [--db-host DB-HOST] [--db-port DB-PORT] [--not-paths NOT-PATHS] [--not-names NOT-NAMES] [--extra-plugins EXTRA-PLUGINS] [--no-init] [--node-version NODE-VERSION]

[NeillM]

I got a full stack trace by adding -vvv to the string passed into the install step:

Exception trace: at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/src/Validate.php:29 MoodlePluginCI\Validate->realPath() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/src/Validate.php:44 MoodlePluginCI\Validate->directory() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/src/Command/InstallCommand.php:149 MoodlePluginCI\Command\InstallCommand->initializeInstallerFactory() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/src/Command/InstallCommand.php:103 MoodlePluginCI\Command\InstallCommand->initialize() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/vendor/symfony/console/Command/Command.php:264 Symfony\Component\Console\Command\Command->run() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/vendor/symfony/console/Application.php:1040 Symfony\Component\Console\Application->doRunCommand() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/vendor/symfony/console/Application.php:301 Symfony\Component\Console\Application->doRun() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/vendor/symfony/console/Application.php:171 Symfony\Component\Console\Application->run() at /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/ci/bin/moodle-plugin-ci:104

[micaherne]

That looks like either the plugin code has been deleted or the working directory has changed. Would you be able to add a couple of steps between the two runs:

sh pwd sh ls -l ./files/converter/onedrive

[NeillM]

I think it is because of the workspace clean up call in the withMoodlePluginCiContainer step.

I'm having a play around with adding a new optional parameter to the step that will stop it being called so my steps would look more like:

withMoodlePluginCiContainer(php: '7.4', db: 'mysql', clean: 'false') {
    ....
}
withMoodlePluginCiContainer(php: '8.1', db: 'postgres') {
    ....
}

[micaherne]

Ah yes, if that's deleting the plugin it's a bug. The intention was just that it would clear up anything it had created, so it shouldn't be deleting stuff that was there before it started.

[NeillM]

Just stopping the workspace clean up from happening does not work, as the step does not want the code there, so the next instance fails anyway.

I guess we need a way of determining which plugin is being tested in that step so we can exclude it, or perhaps fixing the location we check the code out to and then copying it into Moodle during the install stage.

Doing the second option would probably need us to change all the piplelines that use this though

[NeillM]

Or perhaps we move the checkouttosubDirectory command to be inside the moodlePluginCiInstall command

[micaherne]

When you remove the cleanWs step altogether, is it the existence of the "ci" directory that's causing it still to fail when doing multiple runs in the same job? If so, I wonder if it might be worth just explicitly deleting that for the cleanup? I guess that would mean that the calling code would have to make sure it cleaned up the actual plugin and any dependencies though, or it would fail on the next run of the job.

I was hoping to keep the CI library decoupled from the actual provision of the plugin code, so it could support use cases like downloading a third-party plugin from the Moodle plugin database and running the tests on that so I'm not sure about having it check out the code. Also, where a plugin has dependencies there's a step needed to fetch them between checking out the plugin and running the CI - we have a script we run to do this - so it could get quite complicated.

I'm not really sure what the answer is here as you can probably tell!

[NeillM]

Hopefully now I'm back from Christmas I'll get some more time to have a poke around with it.

[NeillM]

I have finally made some time to look at this again and think I have made some progress. So far I have:

1. Split the clean workspace part into it's own command that is optional to use as part of withMoodlePluginCiContainer
2. Added the ability to give withMoodlePluginCiContainer a tag
3. When a tag is present everything is run inside a subdirectory of that name
4. The tag is also included in the container name

This means that when you want to run multiple containers you use a file that looks like:

pipeline {
    options {
        checkoutToSubdirectory('files/converter/onedrive')
   }
  stages {
      stage() {
         steps {
            withMoodlePluginCiContainer(php: '7.4', db: 'mysql', tag: 'm311-php74-mysql', clean: 'false') {
                ....
            }
            withMoodlePluginCiContainer(php: '8.1', db: 'postgres', tag: 'm401-php81-postgres', clean: 'false') {
                ....
            }
          }
      }
  }
  post {
      always {
          moodlePluginCleanWorkspace()
      }
  }
}

[NeillM]

My changes also make it possible to run tests using a matrix

[NeillM]

Here are some Jenkin's files I have used against a fork of one of our plugins for testing things with this change:

1. Matrix setup
2. Multiple tests in serial
3. Basic test