Code checker warnings are not reflected in the full job status

[NeillM]

In a job configured like:

pipeline {
    agent {label 'testing'}

    options {
        checkoutToSubdirectory('files/converter/onedrive')
        disableConcurrentBuilds()
    }

    stages {
        stage("Plugin CI") {
            steps {
                withMoodlePluginCiContainer(php: '8.1', db: 'mysql') {
                    moodlePluginCiInstall("--branch MOODLE_401_STABLE --plugin ./files/converter/onedrive")

                    moodlePluginCi 'phplint'
                    moodlePluginCi 'phpcpd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpmd', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'codechecker --max-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpdoc', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'validate', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'savepoints', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'mustache', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'grunt --max-lint-warnings 0', 'SUCCESS', 'SUCCESS'
                    moodlePluginCi 'phpunit'
                }
            }
        }
    }
}

The linting stage had two warnings in it

moodle-plugin-ci codechecker --max-warnings 0 RUN Moodle CodeSniffer standard on fileconverter_onedrive ...WW..... 10 / 10 (100%)

FILE: /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/moodle/files/converter/onedrive/tests/converter_test.php

FOUND 0 ERRORS AND 1 WARNING AFFECTING 1 LINE

48 | WARNING | Test method test_supports() is missing any coverage information, own or at class level | | (moodle.PHPUnit.TestCaseCovers.Missing)

FILE: /var/lib/jenkins/workspace/leconverter_onedrive_jenkinstest/moodle/files/converter/onedrive/tests/privacy/privacy_test.php

FOUND 0 ERRORS AND 1 WARNING AFFECTING 1 LINE

43 | WARNING | Test method test_get_contexts_for_userid() is missing any coverage information, own or at class level | | (moodle.PHPUnit.TestCaseCovers.Missing)

Time: 386ms; Memory: 18MB

However the overall status of the job was a pass, I would expect it to show up in the warning state.

[NeillM]

The same happens if I deliberately add a coding style failure into the code base.

I'm guessing it is possible that other parts could also have failures such as the unit or behat tests and then they would not cause a change in the status of the overall job.

[NeillM]

Although having looks at my config again just now, I'm wondering if that is my own fault for adding success twice in the commands.

[micaherne]

Yes that sounds like it's because of the configuration. If you add explicit statuses they override the actual stage result and build result. (That was implemented as it's often useful to run some of the code checking stuff to see the results but only fail the build if actual tests fail, or whatever.)

[NeillM]

I can confirm that if I change the setup to:

pipeline {
    agent {label 'testing'}

    options {
        checkoutToSubdirectory('files/converter/onedrive')
        disableConcurrentBuilds()
    }

    stages {
        stage("Plugin CI") {
            steps {
                withMoodlePluginCiContainer(php: '8.1', db: 'mysql') {
                    moodlePluginCiInstall("--branch MOODLE_401_STABLE --plugin ./files/converter/onedrive")

                    moodlePluginCi 'phplint'
                    moodlePluginCi 'phpcpd', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'phpmd', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'codechecker --max-warnings 0', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'phpdoc', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'validate', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'savepoints', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'mustache', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'grunt --max-lint-warnings 0', 'UNSTABLE', 'FAILURE'
                    moodlePluginCi 'phpunit'
                }
            }
        }
    }
}

Then the overall job will show up as unstable (i.e. warning yellow) when the liniting has either an error or a warning, while still running all the other stages in the job. Which is what I would want.

So it was defiantly a me issue.