jqwik-team / jqwik

Property-Based Testing on the JUnit Platform
http://jqwik.net
Eclipse Public License 2.0
578 stars 64 forks source link

test: add IBM Semeru Java distribution to test matrix #481

Open vlsi opened 1 year ago

vlsi commented 1 year ago

Overview

Semeru uses Open9J JIT, so it might behave differently from the other OpenJDK distributions, so it is assigned a greater weight in the test matrix.

The option for testing with "same hashcode" -XX:hashCode=2 is moved to Test task execution only since javac, and kotlinc do not behave well with "same hashcode".

Semeru does not support "same hashcode" option yet, and I raised https://github.com/eclipse-openj9/openj9/issues/17309 to track that.

See https://www.ibm.com/support/pages/semeru-runtimes-getting-started


I hereby agree to the terms of the jqwik Contributor Agreement.

jlink commented 1 year ago

@vlsi It's not ready for merge I assume.

vlsi commented 1 year ago

I guess something wrong went with cache restoration. I'll look into that later. Apparently, PRs with failing CI jobs are not mergeable.

vlsi commented 1 year ago

I'm adding toolchains to Apache JMeter (see https://github.com/apache/jmeter/pull/5989), and it turns out there's a need for configuring several Java versions: 1) Java for build purposes 2) Java for test purposes

I think it might be worth compiling with a fixed Java version only, and then execute tests with multiple Java versions.

At the end of the day, people rarely recompile jqwik, so it sounds reasonable to compile once.

It might make it reasonable to add tasks like testOn11, testOn17, and so on. It would enable compiling once, and executing tests for several Java versions within a single CI job.

WDYT?

jlink commented 1 year ago

I'm adding toolchains to Apache JMeter (see apache/jmeter#5989), and it turns out there's a need for configuring several Java versions:

1. Java for build purposes

2. Java for test purposes

I think it might be worth compiling with a fixed Java version only, and then execute tests with multiple Java versions.

At the end of the day, people rarely recompile jqwik, so it sounds reasonable to compile once.

It might make it reasonable to add tasks like testOn11, testOn17, and so on. It would enable compiling once, and executing tests for several Java versions within a single CI job.

WDYT?

I think it would make sense if the OS/JDK used for compilation was the one I use for release versions. Which would force me to move releasing to GH actions which I'm a bit hesitant about.

I'm also wondering if the randomized build matrix is still pulling its weight. It revealed a TZ-related bug in the beginning; currently it seems to create more trouble than worth.

vlsi commented 1 year ago

I think it would make sense if the OS/JDK used for compilation was the one I use for release versions

Imagine something like

Then it would verify if the code is really consumable with Java 1.8. For instance, the proper way to build with Java 9+ and target 8 is to use --release 8 rather than targetCompatibilty (see https://www.morling.dev/blog/bytebuffer-and-the-dreaded-nosuchmethoderror/ )

Kotlin supports -Xjdk-release=1.8 so it selects proper overloads when using a newer JDK to compile for past releases. If you do not pass -Xjdk-release option, then it would happily use overloads that do not exist in Java 1.8, and it would cause NoSuchMethodError.


it seems to create more trouble than worth.

What are the troubles by the way?