search

eclipse-jdt / eclipse.jdt.core

Eclipse Public License 2.0
141 stars 107 forks source link

[test] Make jdt.core tests configurable to run only on LTS versions #2418

Open jarthana opened 3 weeks ago

jarthana commented 3 weeks ago

By now it's obvious that jdt.core runs far too many tests every time we build. Even though the jenkins run tests only at specific levels, the builds run all of them. Every new test added (even if it is designed to run only from 1.8) will be run 15 times.

I am wondering if we should, at some point, just run only the LTS levels by passing a flag (or even by default). This is me thinking aloud. So, there could be reasons why this may not work.

iloveeclipse commented 3 weeks ago

We run on Jenkins this now 1.8,11,17,21,22 : https://github.com/eclipse-jdt/eclipse.jdt.core/blob/565cabe5b1f680bf0d910231f626ee320f3f8392/Jenkinsfile#L38

Do you mean to have same for the IDE-started tests?

jarthana commented 3 weeks ago

I am thinking about hard-wiring this (logic and versions) in our test framework rather than having to pass versions from outside (like the jenkins file). For e.g., anyone writing a new test suite might simply say, this test runs only on LTS levels and then it does just that. That way, we can shift our testing policy to test-lts-only mode in future.

As I am typing, I am beginning to understand there could be limitations - in case of tests that are introduced since the last LTS. But I guess we can figure out a way.

akurtakov commented 3 weeks ago

LTS versions plus latest version just like done in Jenkinsfile (1.8,11,17,21,22) is the best to hardcode IMO.