search

adoptium / aqa-tests

Home of test infrastructure for Adoptium builds
https://adoptium.net/aqavit
Apache License 2.0
129 stars 308 forks source link

Gather TAP/metadata files for April Temurin builds (for marketplace) #3640

Closed smlambert closed 2 years ago

smlambert commented 2 years ago

In prep for first listing in the Adoptium marketplace, we should grab the TAP and metadata files from the April release. Since the runs did not go very cleanly during the release, it will likely be easier to rerun (once we backport excludes to v0.9.1) using the AQA Test pipeline job.

We may want to augment the gathering of the TAP files from a parent job beyond what the recent changes do for pushing to artifactory (or alternatively, set up an artifactory instance to push to).

We will ultimately want to store a zip file containing all of the TAP and metadata files in the releases repository alongside the binaries (for releases only, not for EA/nightly builds).

JDK version JDK platform link to results files
17 all
17 x86-64_linux tap_jdk17_xlinux_temurin.zip
17 x86-64_mac tap_jdk17_x64mac_temurin.zip
17 x86-64_windows tap_jdk17_x64windows_temurin.zip
17 ppc64le_linux tap_jdk17_ppc64leLinux_temurin.zip
17 s390x_linux tap_jdk17_s390xLinux_temurin.zip
17 aarch64_mac tap_jdk17_aarch64Mac_temurin.zip
17 aarch64_linux tap_jdk17_aarch64Linux_temurin.zip
17 arm_linux
17 x86-32_windows
17 x86-64_alpine-linux
17 ppc64_aix
11 all
11 x86-64_linux tap_jdk11_xlinxu_temurin.zip
11 x86-64_mac tap_jdk11_x64mac_temurin.zip
11 x86-64_window tap_jdk11_x64windows_temurin.zip
11 ppc64le_linux tap_jdk11_ppc64leLinux_temurin.zip
11 aarch64_mac tap_jdk11_aarch64_mac_temurin.zip
11 s390x_linux issues running sanity.perf, unable to rerun due to infra/2578
11 aarch64_linux tap_jdk11_aarch64Linux_temurin.zip
11 arm_linux tap_jdk11_armLinux_temurin.zip
11 x86-32_windows
11 x86-64_alpine-linux
11 ppc64_aix
8 all
8 x86-64_linux tap_jdk8_xlinux_temurin.zip
8 x86-64_mac tap_jdk8_x64mac_temurin.zip
8 x86-64_windows tap_jdk8_x64windows_temurin.zip
8 ppc64le_linux tap_jdk8_ppc64le_linux_temurin.zip
18 all
18 x86-64_linux tap_jdk18_xlinux_temurin.zip
18 x86-64_mac tap_jdk18_x64mac_temurin.zip
18 x86-64_windows tap_jdk18_x64windows_temurin.zip
smlambert commented 2 years ago

I will start with the 3 primary jdk17 platforms:

Pulling builds from https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk17-pipeline/348 and running tests from: AQA_Test_Pipeline/17.

sophia-guo commented 2 years ago

Build failed. sdk_resource=customized, sdk_url

https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/aqaTestPipeline.groovy#L84-L85 might not work @llxia

smlambert commented 2 years ago

https://github.com/adoptium/aqa-tests/issues/3649 - raised as the problem we encounter for Temurin builds.

I also created https://github.com/adoptium/aqa-tests/issues/3650 (but do not think it is a real problem, as I need to add those parameters to the Jenkins config and try using those params instead).

llxia commented 2 years ago

PR Support nested zip file needs to be backported into v0.9.1-release

smlambert commented 2 years ago

Primaries for jdk17:

jdk17 x64 mac (via AQA_Test_Pipeline/23 and Grinder/4635): tap_jdk17_x64mac_temurin.zip 👍

jdk17 x64 linux (via AQA_Test_Pipeline/21 and Grinder/4636 and Grinder/4638): tap_jdk17_xlinux_temurin.zip 👍

jdk17 x64 windows (via AQA_Test_Pipeline/50 and Grinder/4749, Grinder/4750, Grinder/4751): tap_jdk17_x64windows_temurin.zip 👍

sophia-guo commented 2 years ago

Primaries for jdk11:

jdk11 x64 mac ( via AQA_Test_Pipeline/26 and Grinder/4662, Grinder/4663, Grinder/4664):tap_jdk11_x64mac_temurin.zip 👍

jdk11 x64 linux ( via AQA_Test_Pipeline/25 and Grinder/4640 ): tap_jdk11_xlinxu_temurin.zip 👍

jdk11 x64 windows ( via AQA_Test_Pipeline/44 and Grinder/4734, Grinder/4735 ): tap_jdk11_x64windows_temurin.zip 👍

sophia-guo commented 2 years ago

Primaries for jdk18:

jdk18 x64 mac ( via AQA_Test_Pipeline/33 and Grinder/4700 , Grinder/4701): tap_jdk18_x64mac_temurin.zip 👍

jdk18 x64 linux ( via AQA_Test_Pipeline/27 and Grinder/4703 , Grinder/4702): tap_jdk18_xlinux_temurin.zip 👍

jdk18 x64 windows ( via AQA_Test_Pipeline/45 and Grinder/4752, Grinder/4753 ): tap_jdk18_x64windows_temurin.zip 👍

sophia-guo commented 2 years ago

Windows zip file empty issue https://github.com/adoptium/aqa-tests/issues/3668

sophia-guo commented 2 years ago

Primaries for jdk8:

jdk8 x64 mac ( via AQA_Test_Pipeline/46 and Grinder/4706, Grinder/4707, Grinder/4708): tap_jdk8_x64mac_temurin.zip 👍

jdk8 x64 linux ( via AQA_Test_Pipeline/47 and Grinder/4709 , Grinder/4710, Grinder/4732, Grinder/4733 ): tap_jdk8_xlinux_temurin.zip 👍

jdk8 x64 windows ( via AQA_Test_Pipeline/48 and Grinder/4754, Grinder/4755): tap_jdk8_x64windows_temurin.zip 👍

sophia-guo commented 2 years ago

Done. Close.

smlambert commented 2 years ago

Thanks @sophia-guo ! This is great!

I am going to re-open and do the same exercise for the secondary and 3rd tier platforms as I suspect some of them will not need much intervention. Getting the primaries completed is a big relief though, as now we will for sure have a Temurin listing in the marketplace on launch day. 😄

smlambert commented 2 years ago

Secondary platforms, jdk17: aarch64_linux,aarch64_mac,ppc64le_linux,s390x_linux: AQA_Test_Pipeline/52/

jdk17 aarch64_linux: tap_jdk17_aarch64Linux_temurin.zip 👍

jdk17 aarch64_mac: tap_jdk17_aarch64Mac_temurin.zip 👍

jdk17 ppc64le_linux: tap_jdk17_ppc64leLinux_temurin.zip 👍

jdk17 s390x_linux: tap_jdk17_s390xLinux_temurin.zip 👍

smlambert commented 2 years ago

Secondary platforms jdk11: AQA Test Pipeline 53

jdk11 aarch64_linux: tap_jdk11_aarch64Linux_temurin.zip 👍

jdk11 aarch64_mac: tap_jdk11_aarch64_mac_temurin.zip - need to rerun system target OAuthTest, but unable to due to https://github.com/adoptium/infrastructure/issues/2578

jdk11 ppc64le_linux: tap_jdk11_ppc64leLinux_temurin.zip 👍

jdk11 s390x_linux: multiple/various failures (fail to clone TKG repo, error fetching aqa-tests repo build611, etc)

smlambert commented 2 years ago

Tertiary platforms jdk17: AQA Test Pipeline 54 SL/May 25: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/60

arm_linux:

ppc64_aix: won't pursue for marketplace launch, as its not been released yet

x86-32_windows:

x86-64_alpine-linux:

smlambert commented 2 years ago

Tertiary platforms jdk11: AQA Test Pipeline 55 SL/May 25: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/61/ - grid view

arm_linux: tap_jdk11_armLinux_temurin.zip

ppc64_aix:

x86-32_windows:

x86-64_alpine-linux:

smlambert commented 2 years ago

jdk8 2/3rd tier platforms: AQA Test Pipeline 56 SL/May 25: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/62/ - grid view

aarch64_linux: ppc64le_linux: ppc64_aix: sparcv9_solaris: x86-64_solaris: x86-64_alpine-linux:

AQA Test Pipeline 57 x86-32_windows:

AQA Test Pipeline 58 arm_linux:

smlambert commented 2 years ago

jdk18 2/3rd tier platforms: AQA Test Pipeline 59 SL/May25: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/63/ - grid view

aarch64_linux: arm_linux: ppc64_aix: ppc64le_linux: s390x_linux: x86-32_windows: x86-64_alpine-linux: aarch64_mac:

steelhead31 commented 2 years ago

Currently gathering/rerunning on openjdk8/solaris/sparcv9

All TAP files gathered for successful test runs for functional_sanity, fuctional_extended & functional_special.

Outstanding Tests Running, Sanity_System, Extended_System & Grinder Run For Failed OpenJDK Sanity (jdk_math_0 & jdk_util_0)

steelhead31 commented 2 years ago

@smlambert here is the zipfile for sparc/solaris/jdk8

Only the 2 failures ( which should be excluded ) outstanding :-

Openjdk_Sanity : 2 Failures ( jdk_util_0 - java/util/Currency/CurrencyTest.java.CurrencyTest & java/util/Currency/ValidateISO4217.java )

Openjdk_Extended : Single failure ( jdk_jfr_0 )

steelhead31 commented 2 years ago

JDK8 Alpine Linux, TAP Files Attached to this issue with the following tests failing

1) Test_openjdk8_hs_sanity.openjdk_x86-64_alpine-linux_testList_2 : Grinder (4889)

22:54:32 FAILED test targets: 22:54:32 jdk_lang_0 22:54:32 jdk_lang_1

Failed Tests : java/lang/ProcessBuilder/Basic.java.Basic java/lang/ProcessBuilder/Basic.java.Basic

Known Issue ( 2385/3603 - Alpine Linux / Prog ) - Should Be Excluded On Alpine Linux / JDK8

2) Test_openjdk8_hs_extended.openjdk_x86-64_alpine-linux_testList_0 : Grinder (4888 / 4895)

12:45:11 FAILED test targets: 12:45:11 jdk_nio_0 12:45:11 jdk_nio_1 12:45:11 jdk_management_0 12:45:11 jdk_management_1

Failed Tests :

sun/management/jmxremote/bootstrap/CustomLauncherTest.java.CustomLauncherTest
sun/management/jmxremote/bootstrap/CustomLauncherTest.java.CustomLauncherTest

Custom Launcher Known Issue : https://github.com/adoptium/aqa-tests/issues/3238 - Should Be Excluded On Alpine Linux / JDK8

java/nio/file/Files/walkFileTree/find.sh.find
java/nio/file/Files/walkFileTree/find.sh.find

Issue https://github.com/adoptium/aqa-tests/issues/3714 has been raised.

tap_jdk8_alpine_linux_x86_x64.zip

steelhead31 commented 2 years ago

Working on JDK8 x86-64_solaris: All done, rerun of extended suite . openjdk required following Solaris X64 Reboot Current Tap files attached, final run of tests completed. tap_jdk8_solaris_x86_x64.zip . Remaining failures in extended suite due to known issues : jdk_tools_0 fails due to https://github.com/adoptium/aqa-tests/issues/3618 hotspot_jre_0 fails due to https://github.com/adoptium/aqa-tests/issues/3048

All other tap files etc in attached zip file.

sophia-guo commented 2 years ago

Only the 2 failures ( which should be excluded ) outstanding :- Openjdk_Sanity : 2 Failures ( jdk_util_0 - java/util/Currency/CurrencyTest.java.CurrencyTest & java/util/Currency/ValidateISO4217.java )

https://github.com/adoptium/aqa-tests/issues/3640#issuecomment-1141264794

@steelhead31 what's the failures of those two jdk_util tests? Is there issue opened so I can use it as a track to exclude them?

steelhead31 commented 2 years ago

Only the 2 failures ( which should be excluded ) outstanding :- Openjdk_Sanity : 2 Failures ( jdk_util_0 - java/util/Currency/CurrencyTest.java.CurrencyTest & java/util/Currency/ValidateISO4217.java )

#3640 (comment)

@steelhead31 what's the failures of those two jdk_util tests? Is there issue opened so I can use it as a track to exclude them?

@sophia-guo I think it was related to this issue, being fixed in JDK11.. https://github.com/adoptium/aqa-tests/issues/678 , if that isnt sufficient, let me know and I'll raise a new issue.

sophia-guo commented 2 years ago

Thanks @steelhead31! So did you mean that the failure is because https://bugs.openjdk.org/browse/JDK-8150324 wan't backport to jdk8? And this only happen to solaris jdk8?

steelhead31 commented 2 years ago

@sophia-guo thats how I read it yes.

sophia-guo commented 2 years ago

I have rerun tests java/util/Currency/CurrencyTest.java.CurrencyTest & java/util/Currency/ValidateISO4217.java to double check the failures. Unfortunately , both passed. https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4969/ https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4968/

10x https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4976/ https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/4977/

steelhead31 commented 2 years ago

Well done, I tried a few times, and couldnt get them to pass, must be an intermittent thing, or resource issue when running the full set, rather than an individual test... I've had a look at those grinder runs, so I can get the paramters to try myself on a future run :)

sophia-guo commented 2 years ago

https://github.com/adoptium/aqa-tests/issues/3640#issuecomment-1135026844

Tertiary platforms jdk17: AQA Test Pipeline 54 SL/May 25: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/60

arm_linux: Test_openjdk17_hs_sanity.system_arm_linux #163 completed with status FAILURE (propagate: false to ignore) : not available any more Test_openjdk17_hs_sanity.functional_arm_linux #132 completed with status FAILURE (propagate: false to ignore): not available any more Test_openjdk17_hs_special.functional_arm_linux #49 completed with status FAILURE (propagate: false to ignore)

Running on [test-sxa-armv7l-ubuntu2004-odroid-1](https://ci.adoptopenjdk.net/computer/test-sxa-armv7l-ubuntu2004-odroid-1/) in /ssd/jenkins/workspace/Test_openjdk17_hs_special.functional_arm_linux
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 1 hr 0 min
[Pipeline] {
[Pipeline] cleanWs
[Pipeline] }
[Pipeline] // timeout
[Pipeline] echo
Exception: java.nio.file.AccessDeniedException: /ssd/jenkins

Test_openjdk17_hs_sanity.perf_arm_linux #133 completed with status FAILURE (propagate: false to ignore): not available any more Test_openjdk17_hs_extended.perf_arm_linux #49 completed with status FAILURE (propagate: false to ignore)

[Pipeline] node
Running on [test-sxa-armv7l-ubuntu2004-odroid-1](https://ci.adoptopenjdk.net/computer/test-sxa-armv7l-ubuntu2004-odroid-1/) in /ssd/jenkins/workspace/Test_openjdk17_hs_extended.perf_arm_linux
[Pipeline] {
[Pipeline] timeout
Timeout set to expire in 1 hr 0 min
[Pipeline] {
[Pipeline] cleanWs
[Pipeline] }
[Pipeline] // timeout
[Pipeline] echo
Exception: java.nio.file.AccessDeniedException: /ssd/jenkins

Test_openjdk17_hs_extended.functional_arm_linux #132 completed with status FAILURE (propagate: false to ignore): not available any more Test_openjdk17_hs_extended.system_arm_linux #135 completed with status FAILURE (propagate: false to ignore): not available any more Test_openjdk17_hs_sanity.openjdk_arm_linux #132 completed with status FAILURE (propagate: false to ignore): not available any more Test_openjdk17_hs_extended.openjdk_arm_linux #49 completed with status FAILURE (propagate: false to ignore)

Three child jobs https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_arm_linux_testList_0/43/ https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_arm_linux_testList_2/43/ : same failure with test-sxa-armv7l-ubuntu2004-odroid-1 https://ci.adoptopenjdk.net/job/Test_openjdk17_hs_extended.openjdk_arm_linux_testList_1/43/ : same failure with test-sxa-armv7l-ubuntu2004-odroid-1

As most of jobs are not available any more and test-sxa-armv7l-ubuntu2004-odroid-1 looks Ok now ( are running with other jobs succefully) rerun jdk17 arm32 pipeline https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/81/, artifacts unavailable.

rerun with url fix https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/83/console

sophia-guo commented 2 years ago

Trying rerun jdk17 arm32 aqa-test with jdk17 build pipeline and looks like artifacts are not available any more. Take aarch64 as example, which works before and now not. https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk17-pipeline/348/artifact/target/linux/aarch64/temurin/.tar.gz/zip*/temurin.zip

I remember the artifacts of release job will be kept some kinds of forever instead of only last 60 number of builds? @sxa any comments?

sophia-guo commented 2 years ago

jdk11 windows32 https://github.com/adoptium/aqa-tests/issues/3640#issuecomment-1135031367

Only 3 jobs are available and all of them failed with 

09:06:28  + ./get.sh -s /cygdrive/j/jenkins/workspace/Test_openjdk11_hs_special.functional_x86-32_windows/aqa-tests/.. -p x86-32_windows -r customized -j 11 -i hotspot -c 'https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/1910/artifact/target/windows/x32/temurin/*.zip/*zip*/temurin.zip'
09:06:28  TESTDIR: /cygdrive/j/jenkins/workspace/Test_openjdk11_hs_special.functional_x86-32_windows/aqa-tests
09:06:28  load ./testenv/testenv.properties
09:06:28  get jdk binary...
09:06:28  _ENCODE_FILE_NEW=UNTAGGED curl -OLJSks --user ****:**** https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk11-pipeline/1910/artifact/target/windows/x32/temurin/*.zip/*zip*/temurin.zip
09:06:28  temurin.zip is a nested zip
09:06:28  [temurin.zip]
09:06:28    End-of-central-directory signature not found.  Either this file is not

The artifacts url is incorrect.

sophia-guo commented 2 years ago

x86-64_alpine https://github.com/adoptium/aqa-tests/issues/3640#issuecomment-1155338974 sanity.openjdk is not available any more https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_sanity.openjdk_x86-64_alpine-linux/90/

rerun: https://ci.adoptopenjdk.net/view/Test_grinder/job/AQA_Test_Pipeline/82/

extended.openjdk sun/management/jmxremote/bootstrap/CustomLauncherTest.java.CustomLauncherTest, https://ci.adoptopenjdk.net/job/Test_openjdk11_hs_extended.openjdk_x86-64_alpine-linux_testList_1/26/ same as release triage: https://github.com/adoptium/aqa-tests/issues/3596#issuecomment-1105909655

sxa commented 2 years ago

Trying rerun jdk17 arm32 aqa-test with jdk17 build pipeline and looks like artifacts are not available any more. Take aarch64 as example, which works before and now not. https://ci.adoptopenjdk.net/job/build-scripts/job/openjdk17-pipeline/348/artifact/target/linux/aarch64/temurin/*.tar.gz/zip/temurin.zip

I remember the artifacts of release job will be kept some kinds of forever instead of only last 60 number of builds? @sxa any comments?

Now recovered. Apologies for that - I think it was user error on my part.

sophia-guo commented 2 years ago

We will move to July releases and close this one.