JenkinsPipelineSpecification messes with JenkinsRule harness tests

[sdoeringNew]

Desired Behavior

If a JenkinsPipelineSpecification test runs before a test with the JenkinsRule that latter test will fail with an unspecific NullPointerException about an User:

java.lang.NullPointerException
    at hudson.model.User.clear(User.java:737)
    at org.jvnet.hudson.test.JenkinsRule.before(JenkinsRule.java:390)
    at org.jvnet.hudson.test.JenkinsRule$1.evaluate(JenkinsRule.java:595)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
    at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.lang.Thread.run(Thread.java:748)

To workaround this issue it is necessary to restore the Jenkins.HOLDER that the JenkinsPipelineSpecification has overwritten.

Benefits

With this change it would be possible to run both JenkinsPipelineSpecification and JenkinsRule tests.

[sdoeringNew]

I don't see this as a bug because the use cases for this are pretty small.

The solution would be to put this:

    // Restore the Jenkins.HOLDER
    // It was overwritten in: JenkinsPipelineSpecification#setup()
    Jenkins.HOLDER = new Jenkins.JenkinsHolder() {
        Jenkins getInstance() { Jenkins.theInstance }
    }

in the cleanup() of the JenkinsPipelineSpecification.

[sdoeringNew]

In case someone stumbles upon this issue as well I've workarounded this issue by adding a second ClassRule before the JenkinsRule that restores the Jenkins.HOLDER like that:

@Shared
@ClassRule
CleanJenkinsHolderRule cleanJenkinsHolderRule = new CleanJenkinsHolderRule()

@Shared
@ClassRule
JenkinsRule jenkinsRule = new JenkinsRule()

static class CleanJenkinsHolderRule implements TestRule {
    Statement apply(Statement base, Description description) {
        // Restore the Jenkins.HOLDER
        // It was overwritten in: JenkinsPipelineSpecification#setup()
        Jenkins.HOLDER = new Jenkins.JenkinsHolder() {
            Jenkins getInstance() { Jenkins.theInstance }
        }

        return { base.evaluate() } as Statement
    }
}

Of course you could add a cleanup() to every JenkinsPipelineSpecification and restore the Jenkins.HOLDER there.

[awittha]

I like the rule-based solution (workaround?), given that the problem is introduced by using a JenkinsRule in the first place.

I'll file this as FAQ, and hope to eventually include it in wiki-based documentation when we move out of "just Groovydoc."