Open rsvoboda opened 1 month ago
Usability and OOTB experience should be the primary goal for us, especially when we want others to look into our stuff
if I create a reproducer, I simply add -Dreruns=0
.
There is already parametrization for surefire and failsafe available:
it doesn't work for overriding. Both me and @fedinskiy tried it. I didn't try it for a scenario where reruns are not present in the POM.xml
, but I am not in a favor of repetition in Jenkins jobs. Anyway I don't have a strong opinion, so let's go with whatever folk wants.
I recently thought about it and I suggesting this:
reruns
to zero in pomreruns
in Jenkins scripts to the desired number (eg 2 for OCP jobs and 3 for baremetal)I did a small experiment and passing failsafe.rerunFailingTestsCount
works as expected
mvn -am -pl examples/restclient clean verify
mvn -am -pl examples/restclient clean verify -Dfailsafe.rerunFailingTestsCount=5
Code changes:
diff --git a/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java b/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
index d4178679..0c04b82a 100644
--- a/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
+++ b/examples/restclient/src/test/java/io/quarkus/qe/PingPongResourceIT.java
@@ -17,6 +17,6 @@ public class PingPongResourceIT {
@Test
public void shouldPingPongWorks() {
- ping.given().get("/ping/pong").then().statusCode(HttpStatus.SC_OK).and().body(is("pingpong"));
+ ping.given().get("/ping/pong").then().statusCode(HttpStatus.SC_OK).and().body(is("pingpongX"));
}
}
diff --git a/pom.xml b/pom.xml
index 4e95f52c..a2f09c8f 100644
--- a/pom.xml
+++ b/pom.xml
@@ -68,7 +68,6 @@
<infinispan-legacy.image>docker.io/infinispan/server:13.0</infinispan-legacy.image>
<!-- TODO use official image if this fixed https://github.com/hashicorp/docker-consul/issues/184 -->
<consul.image>docker.io/bitnami/consul:1.18.1</consul.image>
- <reruns>2</reruns>
<flaky-run-reporter.version>0.1.2.Beta1</flaky-run-reporter.version>
<certificate-generator.version>0.6.0</certificate-generator.version>
</properties>
@@ -233,7 +232,6 @@
<configuration>
<!-- fixes unavailable native binary build logs, https://github.com/quarkus-qe/quarkus-test-framework/issues/785 -->
<forkNode implementation="org.apache.maven.plugin.surefire.extensions.SurefireForkNodeFactory" />
- <rerunFailingTestsCount>${reruns}</rerunFailingTestsCount>
</configuration>
</plugin>
<plugin>
We can do this to make sure reruns are triggered on GitHub Actions CI
diff --git a/.github/mvn-settings.xml b/.github/mvn-settings.xml
index 32668345..daa5a051 100644
--- a/.github/mvn-settings.xml
+++ b/.github/mvn-settings.xml
@@ -4,6 +4,10 @@
<profiles>
<profile>
<id>google-mirror-jboss-proxy</id>
+ <properties>
+ <failsafe.rerunFailingTestsCount>2</failsafe.rerunFailingTestsCount>
+ <surefire.rerunFailingTestsCount>2</surefire.rerunFailingTestsCount>
+ </properties>
<repositories>
<repository>
<id>google-maven-central</id>
The same can be done for Jenkins based settings file, it would be global settings though.
Parametrization of
rerunFailingTestsCount
is not intuitive and user friendly, these are my concerns:reruns
- https://github.com/quarkus-qe/quarkus-test-framework/blob/main/pom.xml#L71There is already parametrization for surefire and failsafe available:
I would rather see those params defined in CI jobs or in separate profile used in our CI. The same applies to TS where we have 2 parameters ...
CC @michalvavrik @fedinskiy @mjurc