@Order annotation is ignored by Maven but honored in IntelliJ IDEA

[mitch-haraldsson]

Version: 5.7.1

        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.7.1</version>
        </dependency>

Steps to reproduce

Create a test class:

import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderingTest {

    @Test
    @Order(2)
    public void testIGoNext() {
        System.out.println("Test #2");
    }

    @Test
    @Order(3)
    public void testMeIsTheMiddle() {
        System.out.println("Test #3");
    }

    @Test
    @Order(1)
    public void testRunMeFirst() {
        System.out.println("Test #1");
    }

    @Test
    @Order(4)
    public void testSecondToLast() {
        System.out.println("Test #4");
    }

    @Test
    @Order(5)
    public void testTrailing() {
        System.out.println("Test #5");
    }
}

Context

• Used versions (Jupiter/Vintage/Platform): Jupiter Engine 5.7.1
• Build Tool/IDE: IntelliJ IDEA 2020.3.3 / 2021.1

Description

If this code is executed from within IDEA (right click -> Debug), then the execution order is OK.

If however, a maven build is triggered, then the execution order is ignored. An arbitrary order seems to be used:

[sbrannen]

Can you please share your Maven configuration -- for example, for the Maven Surefire plugin?

[mitch-haraldsson]

Sure, I just used a very basic plain pom for testing and was able to recreate the issue:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>JunitOrderingTest</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>13</maven.compiler.source>
        <maven.compiler.target>13</maven.compiler.target>
    </properties>
    <dependencies>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.7.1</version>
        </dependency>
    </dependencies>
</project>

[sbrannen]

It looks like you're missing the Surefire configuration.

Does it work as expected if you add the following to your POM?

    <build>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
            </plugin>
            <plugin>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>
        </plugins>
    </build>

For further details, consult the User Guide.

[mitch-haraldsson]

That fixed it. I'll be darned. Thank you! How is it that it works without surefire plugin, but in the wrong order?

Is there something like an older built-in plugin which does not honor certain configurations in Maven?

[sbrannen]

That fixed it. I'll be darned. Thank you!

You're welcome.

How is it that it works without surefire plugin, but in the wrong order?

I imagine that's a question that many people have asked over the years.

I got hit by it once, too, many many years ago.

In summary, it sounds like your tests were being executed as POJO Tests instead of JUnit tests.