adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
132 stars 312 forks source link

Could we have spotted the missing `jvmti` support in JDK19 on non-x64/aarch64 platforms? #4006

Open sxa opened 2 years ago

sxa commented 2 years ago

Under some of the testing performed during the release cycle for JDK19 we discovered that the --enable-preview option resulted in jvmti support not being available on certain platforms (Notably the ones where a full Virtual Threads implementation was not available - everything except the x86-64 and aarch64 platforms).

Could we have pre-emptively caught this as part of the regular test cycles in order to avoid hitting the problem during the release cycle, and if not is there anything we could put in place to get early warning of such a problem in the future?

smlambert commented 2 years ago

The answer is yes, if someone had been triaging the jdk19 pipelines, those failures were present in nightly and weekly runs.

Example: https://trss.adoptium.net/output/test?id=63130ae993ac60470b274021

[2022-09-03T04:15:39.295Z] Exception in thread "main" java.lang.UnsupportedOperationException: Unsupported os, arch, or address size: Linux, s390x, 64
[2022-09-03T04:15:39.295Z]  at java.base/jdk.internal.foreign.CABI.current(CABI.java:69)
[2022-09-03T04:15:39.295Z]  at java.base/jdk.internal.foreign.abi.SharedUtils.getSystemLinker(SharedUtils.java:237)
[2022-09-03T04:15:39.295Z]  at java.base/java.lang.foreign.Linker.nativeLinker(Linker.java:198)
[2022-09-03T04:15:39.295Z]  at ImplicitAttach.main(ImplicitAttach.java:48)

Also if the advisement of the release checklist to run a trial release pipeline a week ahead of the release had been followed, these failures would have been spotted.

Screen Shot 2022-10-04 at 12 52 56 PM

Looks like it was removed from https://github.com/adoptium/adoptium/issues/171.

Closing this issue, as the 'action' to take is to monitor the test pipelines, and follow the release checklist.

Noting that testing is still disabled, so any nightly monitoring will happen once testing is re-enabled, which likely needs to be added to the checklist to remind release champions. (see https://github.com/adoptium/adoptium/issues/177).

sxa commented 2 years ago

Those failures are in the FFI tests which is equally valid although digging into jvmti it looks like they might be excluded somewhere so we wouldn't have picked them up:

Looking at https://github.com/adoptium/aqa-tests/blob/f494e46151a96f625bfb24f12a6cc3241721405b/openjdk/playlist.xml#L189 it looks like hotspot_serviceability which contains the jvmti tests are only being run for JDK8/9 but only a subset (excluding jvmti) are in those logs and I can't see any indication of the serviceability ones being run in the 19 logs. So I guess the questions are

  1. Should the serviceability tests be excluded for later versions? (A trial on JDK19 shows it is a bit messy although even some of the "passing" ones in there seem to be failing when I click on them) - most are complaining about Use -nativepath to specify the location of native code but I guess that may be easy to resolve without too much of a problem - possible related issue.
  2. What is excluding jvmti from the serviceability tests and should that be reversed? (2/4 are listed in ProblemList_openjdk9.txt) but unclear why none are run)
sxa commented 2 years ago

FYI @sophia-guo since we were discussing this earlier. It probably makes sense to either change the title on this issue now or open a new one to cover attempting to re-enable jvmti. While the FFI tests are also failing, those are enabled and running so could have been caught, but not the jvmti ones.

(Of course this doesn't solve the problem which is that they will fail on JDK19 with --enable-preview on!)

References for the upstream bugs to cover enabling Project Loom on various platforms (Thanks @jerboaa for tracking them down for me!):

sophia-guo commented 2 years ago

Enable serviceability_jvmti tests (serviceability/jvmti) for 19+ https://github.com/adoptium/aqa-tests/pull/4035/files. Note those tests are not running with option --enable-preview, which probably should be solved by adding option tests in upstream openjdk repo.

All Loom related targets are as following, which can be gradually enabled when it's ready. tier1_loom = \ :tier1_loom_runtime \ :tier1_loom_serviceability

tier1_loom_runtime = \ runtime/vthread \ runtime/jni/IsVirtualThread

tier1_loom_serviceability = \ serviceability/jvmti/vthread \ serviceability/jvmti/events \ serviceability/dcmd/thread

jerboaa commented 2 years ago

See also: https://bugs.openjdk.org/browse/JDK-8287726 Does aquavit handle @require tags in jtreg?

sophia-guo commented 2 years ago

See also: https://bugs.openjdk.org/browse/JDK-8287726 Does aquavit handle @require tags in jtreg?

Yes, aqavit using jtreg running openjdk tests. All jtreg features are supported by default.