search

eclipse-platform / eclipse.platform.releng.aggregator

Aggregated repository for Eclipse Java IDE
https://www.eclipse.org/eclipse/
Eclipse Public License 2.0
27 stars 67 forks source link

more different verification matrix builds #2114

Open jukzi opened 1 month ago

jukzi commented 1 month ago

I am unhappy with the current github verification builds as defined in https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blame/master/.github/workflows/mavenBuild.yml

Currently we have 3 verification builds that run on the same java version (21) on 3 different OS in a matrix. Java 17 is not tested. The API test needlessly runs on every OS - but should be OS independent (isn't it?) After running all junit tests, which takes much time, the plugin versions are verified. When that version cheks fais all junit test results are lost - not visible.

Would it not better to have a List of Testscenarios that run like: a) API & version checks only (fail fast), linux (fastest OS) + java 22 (i.e latest JDK) b) junit only - on WIN+ java 17 (i.e. earliest supported JDK) c) junit only - on linux+ java 21, (i.e. latest LTS JDK) d) junit only - on MacOs + java 22 (i.e. latest JDK)

that way we should see missing version bumps faster, see test result even if other tests fail. WDYT?

HannesWell commented 1 month ago

Using different JDKs for different OS makes it more difficult to identify if a failure is an OS or a JDK version problem. If we had infinite resources the full matrix combining each OS with each relevant JDK should run

Having a fast path for version and API checks sounds good. But beware that although the API is often OS independent, it is not always. For example in SWT the API has to be checked for each OS since the code is in the fragments.

But if the configuration is chosen smart it can probably be done with little extra changes.

laeubi commented 1 month ago

Currently we have 3 verification builds that run on the same java version (21) on 3 different OS in a matrix. Java 17 is not tested.

Java 21 is the Java version Eclipse ships with, so what would be the advantage to test with Java 17 (or any other java), this will just increase the runtime and resources, are there any concrete issues you want to address? I once setup such matrix for SWT (on the Java 11 > 17 transition) but it has brought little value. Also as soon as platform chooses to raise the limit for one bundle or any library uses java 21 this won't work so we really should go forward.

Beside that, if required, individual projects can setup additional checks.

The API test needlessly runs on every OS - but should be OS independent (isn't it?)

It depends, e.g. SWT can have OS specific APIs (see Windows OLE support)

After running all junit tests, which takes much time, the plugin versions are verified. When that version cheks fais all junit test results are lost - not visible.

Can you give an example? The action is marked as "always" so the results will always be collected and should always be published.

WDYT?

In general I'm not convinced any "fast checks" are required, they just burn resources we currently use for free, the fastest way is to simply run the maven build on your local maschine, then you can skip whatever you want, see testfailures in any stage, you can choose to either -ff or -fn or -fae and so on.

A PR is to validate as much as possible it is not meant as a debug facility and we better improve the tooling then, e.g. if m2e nature is enabled, you will see any build error even in the IDE (even though I haven't tested it lately how well they are presented).