Open sxa opened 2 years ago
sxa
commented
2 years ago FYI @andrew-m-leonard @smlambert (Shelley I sent you a log from this a while back and I think there may be fewer such messages now but unclear if that's my memory or if it was a slightly different error!
sxa
commented
2 years ago Also tagging @sophia-guo since she was involved in the discussions on the community call earlier today :-)
sophia-guo
commented
2 years ago According to the discussion if jenkins jobs are triggered with undefined parameters there will be those warnings in jenkins server log? If that is the case those warnings can be avoided by re-generating aqa-tests or smoketests jobs triggered by upstream.
Initially when new features was added there was a period that aqa-tests or smoke tests jobs were triggered with undefined parameters. For example, USE_TESTENV_PROPERTIES, the reason is testJobTemplate was not updated accordingly as it didn't affect the functionalities. I have double checked all parameters of aqa-tests and smoke tests, all of them have been updated in testJobTemplate. That means if there is still those warning message related with aqa-tests or smoketests regenerating those jobs should fix it. Note autoGen is enabled with aqa-tests and disabled with smoketests.
sxa
commented
2 years ago Thanks for checking @sophia-guo! It looks like the remaining ones are mostly on the build pipelines so the test jobs may now be all sorted. I've just done a check on the recent logs and these are the remaining warnings - interestingly only the (new) jdk19u pipelines are showing up here @andrew-m-leonard:
jenkins@jenkins-hetzner-ubuntu2004:~$ gzip -cd /var/log/jenkins/jenkins.log.?.gz | grep Skipped.*undefined /var/log/jenkins/jenkins.log | sed -e 's/.*Skipped/Skipped/g' -e 's/. Set.*/./g' | sort | uniq -c
282 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-aix-ppc64-temurin`.
291 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-alpine-linux-aarch64-temurin`.
306 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-alpine-linux-x64-temurin`.
291 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-aarch64-temurin`.
286 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-arm-temurin`.
278 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-ppc64le-temurin`.
270 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-s390x-temurin`.
294 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-linux-x64-temurin`.
485 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-mac-aarch64-temurin`.
488 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-mac-x64-temurin`.
432 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-windows-x64-temurin`.
447 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk19u/jdk19u-windows-x86-32-temurin`.
36 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/create_installer_mac`.
72 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/sign_installer`.
240 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/sign_temurin_gpg`.
jenkins@jenkins-hetzner-ubuntu2004:~$ 
andrew-m-leonard
commented
2 years ago @sxa I don't understand the above, none of the build jobs have a Parameter called NODE_LABEL, they only have 4 paramaters:
sxa
commented
2 years ago That sounds consistent with the warning message - "it is undefined on
sxa
commented
1 year ago Had a look at the log for something else today. We've still got:
USE_TESTENV_PROPERTIES on most of the test jobs giving the warningRELEASE_TAG showing ONLY on Test_openjdk11_j9_sanity.openjdk_x86-64_linux in the most recenr log. Not sure why that might be special.NODE_LABEL on most of the build jobsGENERATE_JOBS on most of the test jobsDYNAMIC_COMPILE on most of the test jobs.
smlambert
commented
1 year ago Regenerated the following test jobs:
JDK 11,17,19 hotspot/platforms https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/864/ (functional - all levels) https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/865/ (perf,system,openjdk - sanity,extended)
JDK 8 hotspot/platforms https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/866/ (functional - all levels) https://ci.adoptopenjdk.net/view/Test_grinder/job/Test_Job_Auto_Gen/867/ (perf,system,openjdk - sanity,extended)
sxa
commented
1 year ago @smlambert Most of these are looking good now, but Test_openjdk8_dragonwell_sanity.system_aarch64_linux is still throwing messages about GENERATE_JOBS being missing, and that looks to be correct. Can that one be regenerated too?
Simiplery for USE_TESTENV_PROPERTIES on a number of the non-temurin test jobs.
[EDIT: See next comment for why JDK8/dragonwell may well be a special case]
sxa
commented
1 year ago A number of the NODE_LABEL messages are from sign_temurin_gpg which is hard coded to use a label of gpgsign but a parameter for it is still being passed in. Since this shouldn't be a "user-serviceable change" I'm going to remove that from the calling pipeline.
Similarly for create_installer_mac it is hard coded to mac&&macos10.14&&xcode10 and does not take a NODE_LABEL parameter, and sign_installer which is tied to eclipse-codesign
@andrew-m-leonard I suspect there must be other places in there where we're passing in a NODE_LABEL where it's not welcome, for example the jdk20-release-*-temurin jobs are all showing Skipped parameterNODE_LABELas it is undefined when they are called. There are three more occurrences outside the temurin jobs - jdk8u-linux-aarch64-dragonwell (Suspiciously the same build as per the previous comment...), jdk8u-windows-x64-openj9 and jdk-linux-aarch64-hotspot (The one which we continue to build hotspot as well as temurin for a test)
sxa
commented
1 year ago @andrew-m-leonard After merging https://github.com/adoptium/ci-jenkins-pipelines/pull/668 we've still got a few remaining occurrences on the PR tester jobs which will need addressing:
244 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk11u/jdk11u-linux-aarch64-temurin` (#114)
373 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk11u/jdk11u-linux-x64-temurin` (#114)
347 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk11u/jdk11u-mac-x64-temurin` (#115)
174 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk11u/jdk11u-windows-x64-temurin` (#116)
244 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk17u/jdk17u-linux-aarch64-temurin` (#109)
373 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk17u/jdk17u-linux-x64-temurin` (#125)
190 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk17u/jdk17u-mac-x64-temurin` (#109)
94 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk17u/jdk17u-windows-x64-temurin` (#108)
244 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk20/jdk20-linux-aarch64-temurin` (#8)
373 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk20/jdk20-linux-x64-temurin` (#8)
347 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk20/jdk20-mac-x64-temurin` (#8)
174 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk20/jdk20-windows-x64-temurin` (#8)
244 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk8u/jdk8u-linux-aarch64-temurin` (#105)
363 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk8u/jdk8u-linux-x64-temurin` (#105)
190 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk8u/jdk8u-mac-x64-temurin` (#105)
382 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts-pr-tester/build-test/jobs/jdk8u/jdk8u-windows-x64-temurin` (#106)Plus these in other build jobs:
2 Skipped parameter `DYNAMIC_COMPILE` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-windows-x64-openj9_SmokeTests` (#163)
306 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/evaluation/jobs/jdk20/jdk20-evaluation-windows-aarch64-temurin` (#20)
58 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-aix-ppc64-temurin` (#221)
177 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-linux-s390x-temurin` (#199)
159 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-windows-x64-openj9` (#239)
424 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-windows-x64-temurin` (#206)
376 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk17u/jdk17u-windows-x86-32-temurin` (#192)
1 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/release/jobs/jdk20/jdk20-release-aix-ppc64-temurin` (#6)
20 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/release/sign_temurin_gpg` (#14,577)NOTE: Since was the log of the last day the presence of mostly jdk17 jobs is likely because of when that versions was scheduled and should not be assumed to mean that only that versions is affected
sxa
commented
1 year ago @smlambert @sophia-guo This is the remaining list from the test jobs. Interestingly most of these seem to be on JDK17 jobs. A number of the extended.openjdk jobs will have been regenerated as part of https://github.com/adoptium/infrastructure/issues/2893#issuecomment-1492259896 but it looks we've still get quite a lof ot he testList_ jobs.
Here's the full list from the last day's log: Test_skipped_parameter.gz
NOTE: Since was the log of the last day the presence of mostly jdk17 jobs is likely because of when that versions was scheduled and should not be assumed to mean that only that versions is affected
sxa
commented
1 year ago After to fix https://github.com/adoptium/ci-jenkins-pipelines/issues/312 it's looking a lot cleaner :-) We do still have quite a few others but I can raise that in
2023-06-13 10:26:01.984+0000 [id=1607485] WARNING hudson.model.ParametersAction#filter: Skipped parameter `USE_TESTENV_PROPERTIES` as it is undefined on `Test_openjdk8_hs_sanity.system_arm_linux` (#334). Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
2023-06-13 10:26:01.985+0000 [id=1607485] WARNING hudson.model.ParametersAction#filter: Skipped parameter `GENERATE_JOBS` as it is undefined on `Test_openjdk8_hs_sanity.system_arm_linux` (#334). Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.
2023-06-13 10:26:01.985+0000 [id=1607485] WARNING hudson.model.ParametersAction#filter: Skipped parameter `DYNAMIC_COMPILE` as it is undefined on `Test_openjdk8_hs_sanity.system_arm_linux` (#334). Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.Most of the above are on openjdk8 test jobs, although some are still in build although not the temurin variants by the look of it e.g.:
2023-06-13 00:52:30.716+0000 [id=1588198] WARNING hudson.model.ParametersAction#filter: Skipped parameter `DYNAMIC_COMPILE` as it is undefined on `build-scripts/jobs/jdk8u/jdk8u-linux-aarch64-dragonwell_SmokeTests` (#219). Set `-Dhudson.model.ParametersAction.keepUndefinedParameters=true` to allow undefined parameters to be injected as environment variables or `-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]` to whitelist specific parameter names, even though it represents a security breach or `-Dhudson.model.ParametersAction.keepUndefinedParameters=false` to no longer show this message.@smlambert will let me know how to regen the jobs to see if that's what's leaving some of the above messages (unclear if it's the build or test jobs that are required though.
sxa
commented
1 year ago Current status:
DYNAMIC_COMPILE parameterGENERATE_JOBS, DYNAMIC_COMPILE and USE_TESTENV_PROPERTIES and some others. Log of occurences today: file.txt.gzsxa
commented
1 year ago @smlambert @sophia-guo I just checked the logs today and most of the remaining ones seem to be from a small number of jobs - Grinder_testList_0 and some of the JDK17 x64_windows ones. Since it only affects those can they be easily fixed? I presume a regen or somethign shoudl resolve this (can the grinder testlist one just be deleted if it will get regenerated if someone runs a grinder with Dynamic=Parallel?)
| Occurences | parameter | job |
|---|---|---|
| 704 | NON_AQA_TEST_REPOS | Grinder_testList_0 |
| 704 | PLATFORM_AND_MACHINE | Grinder_testList_0 |
| 704 | RERUN_ITERATIONS | Grinder_testList_0 |
| 616 | SETUP_JCK_RUN | Test_openjdk17_hs_extended.functional_x86-64_windows |
| 910 | SETUP_JCK_RUN | Test_openjdk17_hs_extended.perf_x86-64_windows |
| 612 | SETUP_JCK_RUN | Test_openjdk17_hs_extended.system_x86-64_windows |
| 612 | SETUP_JCK_RUN | Test_openjdk17_hs_sanity.functional_x86-64_windows |
| 731 | SETUP_JCK_RUN | Test_openjdk17_hs_sanity.openjdk_x86-64_windows |
| 221 | SETUP_JCK_RUN | Test_openjdk17_hs_sanity.perf_x86-64_windows |
| 876 | SETUP_JCK_RUN | Test_openjdk17_hs_sanity.system_x86-64_windows |
| 703 | SETUP_JCK_RUN | Test_openjdk17_hs_special.functional_x86-64_windows |
| 704 | TEST_IMAGES_REQUIRED | Grinder_testList_0 |
smlambert
commented
1 year ago Correct, we can delete that job, as it will be regenerated automatically next time someone runs a parallel grinder. Alternatively I will actually run a Grinder in parallel with GENERATE_JOBS=true to force those child jobs to be regen-ed.
I have manually regen-ed the Windows jobs, so those should now be clean.
sxa
commented
1 year ago Thanks - I was hoping it would be that simple :-) I'll try and remember to check the logs again tomorrow
sxa
commented
11 months ago Still quite a lot of these:
The others are:
jenkins@jenkins-hetzner-ubuntu2004:/var/log/jenkins$ grep -i skipped.parameter jenkins.log | grep -v openjdk17.*alpine | grep -v openjdk17.*arm_ | cut -d\` -f4 | sort | uniq -c
1 build-scripts/jobs/jdk11u/jdk11u-aix-ppc64-hotspot
2 build-scripts/jobs/jdk11u/jdk11u-linux-aarch64-bisheng_SmokeTests
2 build-scripts/jobs/jdk11u/jdk11u-linux-aarch64-dragonwell_SmokeTests
204 build-scripts/jobs/jdk11u/jdk11u-linux-aarch64-openj9_SmokeTests
229 build-scripts/jobs/jdk17u/jdk17u-linux-aarch64-openj9_SmokeTests
229 build-scripts/jobs/jdk17u/jdk17u-linux-x64-openj9_SmokeTests
4 Test_openjdk11_bisheng_extended.functional_aarch64_linux
4 Test_openjdk11_bisheng_extended.system_aarch64_linux
4 Test_openjdk11_bisheng_sanity.functional_aarch64_linux
174 Test_openjdk11_bisheng_sanity.openjdk_aarch64_linux
4 Test_openjdk11_bisheng_sanity.perf_aarch64_linux
102 Test_openjdk11_bisheng_sanity.system_aarch64_linux
6 Test_openjdk11_dragonwell_extended.functional_aarch64_linux
6 Test_openjdk11_dragonwell_extended.system_aarch64_linux
6 Test_openjdk11_dragonwell_sanity.functional_aarch64_linux
234 Test_openjdk11_dragonwell_sanity.openjdk_aarch64_linux
6 Test_openjdk11_dragonwell_sanity.perf_aarch64_linux
6 Test_openjdk11_dragonwell_sanity.system_aarch64_linux
6 Test_openjdk11_hs_extended.functional_arm_linux
6 Test_openjdk11_hs_extended.system_arm_linux
6 Test_openjdk11_hs_sanity.functional_arm_linux
618 Test_openjdk11_hs_sanity.perf_arm_linux
6 Test_openjdk11_hs_sanity.system_arm_linuxMost popular parameters causing issues are:
smlambert
commented
10 months ago thanks @sophia-guo for https://github.com/adoptium/ci-jenkins-pipelines/pull/883 which will regen test jobs regularly and thus relieving the need to manually run regen, and keep the test jobs up-to-date, therefore reducing skipped warnings from those jobs
sxa
commented
10 months ago which will regen test jobs regularly and thus relieving the need to manually run regen, and keep the test jobs up-to-date, therefore reducing skipped warnings from those jobs
My expectation from last weeks' conversations was that this would have caused a regeneration earlier this week so they would be in sync. Is there a chance that didn't happen? I don't know exactly where the regeneration takes place but it would be good to verify if it did happen, and understand if there is still an underlying issue that needs to be addressed.
sophia-guo
commented
10 months ago Weekly pipeline with parameter releaseType=Weekly will trigger the regen https://ci.adoptium.net/job/build-scripts/job/weekly-openjdk11-pipeline/156/parameters/. The code is in https://github.com/adoptium/ci-jenkins-pipelines/pull/883/files.
The regen didn't happen as the typo. The fix PR https://github.com/adoptium/ci-jenkins-pipelines/pull/907
sxa
commented
10 months ago Just realised I have this in a draft in my browser but hadn't sent it. Based on the runs this week:
We are down to one instance in the build jobs:
1 Skipped parameter `NODE_LABEL` as it is undefined on `build-scripts/jobs/jdk11u/jdk11u-linux-arm-hotspot` (#933).And the rest are all in test jobs, including some in the new _rerun jobs, which might indicate that the generation is still not quite doing what it should. Here is a complete list of what is outstanding (prefixed by a number showing the number of instances in the latest log):
jenkins-skipped-parameters.gz
A lot of them seem to be Bisheng/Dragonwell and the riscv64 ones - can we get those explicitly regenerated?
smlambert
commented
10 months ago I was going to propose that we stop building and testing the jobs that no one is caring for, as part of our cleanup prior to Trestle. This includes openj9, bisheng, dragonwell, corretto, etc.
sxa
commented
10 months ago I was going to propose that we stop building and testing the jobs that no one is caring for, as part of our cleanup prior to Trestle. This includes openj9, bisheng, dragonwell, corretto, etc.
I think I've proposed that in the past but there was a desire to keep them to demonstrate that they still worked. It would be nice in the short term to trigger a regen if possible though (I'm guessing they aren't in the weekly pipelines at present, so likely won't be in the 'trigger on tag' builds that we replace those with Oh they are, but the pipelines are all red for various reasons so it's possible it didn't get as far as the testing[*] - would it run the generator just before running the test jobs?) and persue potential de-scoping of other variants under a separate issue. We could even have a "build grinder" job that people can trigger if desired.
My gut feel is that it's "nice" to try and verify that we haven't broken the ability to build those other projects with our scripts, but we could raise it with the working group, since if test is not interested then there may be no takers. It seems that no-one in build is actively looking at the results of those pipelines at present. It's a shame we can't publicise the pipelines and get volunteers to maintain them.
EDIT: Also noting that given our reduction in Linux/x64 capacity removing builds on that platform would potential be quite welcome at the moment!
[*] - Corretto failed with 17:05:49 configure: error: "7.1" is not a valid numerical value for VERSION_BUILD which we've seen before
sxa
commented
10 months ago @smlambert @sophia-guo Looking at one of the logs for Bisheng it has:
17:13:43 Scheduling project: [Test_Job_Auto_Gen](https://ci.adoptium.net/job/Test_Job_Auto_Gen/)
[Pipeline] build (Building Test_Job_Auto_Gen)
17:13:43 Scheduling project: [Test_Job_Auto_Gen](https://ci.adoptium.net/job/Test_Job_Auto_Gen/)
[Pipeline] build (Building Test_Job_Auto_Gen)
17:13:43 Scheduling project: [Test_Job_Auto_Gen](https://ci.adoptium.net/job/Test_Job_Auto_Gen/)
17:13:45 Starting building: [Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_sanity.system_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.system_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_extended.system_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_extended.system_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_sanity.perf_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.perf_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_sanity.functional_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.functional_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_sanity.external_x86-64_linux #71](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.external_x86-64_linux/71/)
17:13:45 Starting building: [Test_openjdk11_bisheng_extended.functional_x86-64_linux #301](https://ci.adoptium.net/job/Test_openjdk11_bisheng_extended.functional_x86-64_linux/301/)
17:13:45 Starting building: [Test_openjdk11_bisheng_extended.openjdk_x86-64_linux #98](https://ci.adoptium.net/job/Test_openjdk11_bisheng_extended.openjdk_x86-64_linux/98/)
17:13:45 Starting building: [Test_openjdk11_bisheng_extended.perf_x86-64_linux #98](https://ci.adoptium.net/job/Test_openjdk11_bisheng_extended.perf_x86-64_linux/98/)
17:13:45 Starting building: [Test_openjdk11_bisheng_special.functional_x86-64_linux #98](https://ci.adoptium.net/job/Test_openjdk11_bisheng_special.functional_x86-64_linux/98/)
17:13:48 Starting building: [Test_Job_Auto_Gen #1501](https://ci.adoptium.net/job/Test_Job_Auto_Gen/1501/)
17:13:49 Starting building: [Test_Job_Auto_Gen #1503](https://ci.adoptium.net/job/Test_Job_Auto_Gen/1503/)
17:13:49 Starting building: [Test_Job_Auto_Gen #1502](https://ci.adoptium.net/job/Test_Job_Auto_Gen/1502/)
17:15:43 Build [Test_openjdk11_bisheng_sanity.external_x86-64_linux #71](https://ci.adoptium.net/job/Test_openjdk11_bisheng_sanity.external_x86-64_linux/71/) completed: FAILUREWould that indicate that the autogen jobs are potentially getting run in parallel with the executions so the regen may not have completed before the jobs were run? I'm not sure that would fully cover why we're still seeing.
EDIT: Looks like the three generator jobs only covered special.openjdk dev.functional and dev.openjdk - I'm running generators on sanity.openjdk and extended.openjdk to see if that clears things up a bit for that example, but I'm unclear on why those wouldn't have been generated along with the others (Those two hadn't seemingly been modified since 2021):
-rw-r--r-- 1 jenkins jenkins 45692 Jan 29 18:16 Test_openjdk11_bisheng_dev.functional_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 45686 Jan 29 18:16 Test_openjdk11_bisheng_dev.openjdk_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33886 Jun 1 2021 Test_openjdk11_bisheng_extended.functional_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33879 Jun 1 2021 Test_openjdk11_bisheng_extended.openjdk_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 45732 Jan 30 00:35 Test_openjdk11_bisheng_extended.openjdk_x86-64_linux_rerun/config.xml
-rw-r--r-- 1 jenkins jenkins 33978 Jun 5 2021 Test_openjdk11_bisheng_extended.openjdk_x86-64_linux_testList_0/config.xml
-rw-r--r-- 1 jenkins jenkins 33978 Jun 5 2021 Test_openjdk11_bisheng_extended.openjdk_x86-64_linux_testList_1/config.xml
-rw-r--r-- 1 jenkins jenkins 33978 Jun 5 2021 Test_openjdk11_bisheng_extended.openjdk_x86-64_linux_testList_2/config.xml
-rw-r--r-- 1 jenkins jenkins 33849 Jun 12 2021 Test_openjdk11_bisheng_extended.perf_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33853 Jun 12 2021 Test_openjdk11_bisheng_extended.system_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 41489 Apr 2 2022 Test_openjdk11_bisheng_sanity.external_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33884 Jun 1 2021 Test_openjdk11_bisheng_sanity.functional_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33877 Jun 1 2021 Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 45715 Jan 31 16:34 Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux_rerun/config.xml
-rw-r--r-- 1 jenkins jenkins 33847 Jun 12 2021 Test_openjdk11_bisheng_sanity.perf_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33851 Jun 12 2021 Test_openjdk11_bisheng_sanity.system_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 33885 Jun 1 2021 Test_openjdk11_bisheng_special.functional_x86-64_linux/config.xml
-rw-r--r-- 1 jenkins jenkins 45690 Jan 29 18:18 Test_openjdk11_bisheng_special.openjdk_x86-64_linux/config.xmlEDIT: After regeneration the sanity.openjdk config.xml for Bisheng increased in size from 33879 bytes to 45691 so it looks like there was quite a bit missing. The list at the top of https://ci.adoptium.net/job/Test_Job_Auto_Gen/ is notable by the absence of some of the other variants having been included for some reason.
sxa
commented
10 months ago The regen didn't happen as the typo. The fix PR https://github.com/adoptium/ci-jenkins-pipelines/pull/907
Approved, although it looks like quite a few regen jobs did get triggered during the weekly pipeline earlier this week despite this.
sxa
commented
10 months ago EDIT: After regeneration the sanity.openjdk config.xml for Bisheng increased in size from 33879 bytes to 45691 so it looks like there was quite a bit missing. The list at the top of https://ci.adoptium.net/job/Test_Job_Auto_Gen/ is notable by the absence of some of the other variants having been included for some reason.
@sophia-guo is there an easy way to force regeneration of some of these? It's not quite clear to me why only some of them are getting regenerated. It looks like me manually regenerating Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux` has made a difference though, so that's good - I'd just rather not have to run that for all of the problematic test jobs if possible.
sophia-guo
commented
10 months ago EDIT: Looks like the three generator jobs only covered special.openjdk dev.functional and dev.openjdk
Those regens are triggerred as there are no jobs generated before. If you see the regened job https://ci.adoptium.net/job/Test_openjdk11_bisheng_special.openjdk_x86-64_linux/ there is only one build right now.
I'm running generators on sanity.openjdk and extended.openjdk
Those jobs were generated before so won't be regened by default unless set AQA_AUTO_GEN=true. With Weekly regen, they will be regened weekly.
sxa
commented
10 months ago I'm running generators on sanity.openjdk and extended.openjdk
Those jobs were generated before so won't be regened by default unless set AQA_AUTO_GEN=true. With Weekly regen, they will be regened weekly.
Ah, but it hasn't happened on the pipelines at the start of the week because of the weekly/Weekly typo. That explains it - ignore what I wrote below :-) We need to run another weekly pipeline before it forces regen which will resolve it :+1:
Hmmm so can you tell why the Bisheng11 sanity.openjdk was not not regenerated before https://ci.adoptium.net/view/Test_openjdk/job/Test_openjdk11_bisheng_sanity.openjdk_x86-64_linux/301 which was started from the weekly pipeline at https://ci.adoptium.net/job/build-scripts/job/weekly-openjdk11-pipeline/156/ a few days ago? It looks like the openjdk11-pipeline from the weekly job didn't get AQA_AUTO_GEN=true but from what you've said it should have if I understand you correctly. I'm probably missing something really obvious here :-)
sxa
commented
9 months ago The only ones appearing more than once in the latest log are:
190 Skipped parameter `ADDITIONAL_ARTIFACTS_REQUIRED` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_0` (#38).
264 Skipped parameter `ADDITIONAL_ARTIFACTS_REQUIRED` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_1` (#38).
190 Skipped parameter `ADDITIONAL_ARTIFACTS_REQUIRED` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_2` (#38).
190 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_0` (#38).
264 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_1` (#38).
190 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk21_hs_extended.openjdk_x86-64_linux_testList_2` (#38).
2 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk22_hs_extended.openjdk_s390x_linux_testList_0` (#13).
2 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk22_hs_extended.openjdk_s390x_linux_testList_1` (#13).
2 Skipped parameter `BASE_DOCKER_REGISTRY_CREDENTIAL_ID` as it is undefined on `Test_openjdk22_hs_extended.openjdk_s390x_linux_testList_2` (#13).
31 Skipped parameter `RERUN_ITERATIONS` as it is undefined on `Test_openjdk17_hs_sanity.external_x86-64_alpine-linux` (#10).I'm tempted to just remove the testList ones and let them be regenerated when required. It looks like the last one referencing RERUN_ITERATIONS has not yet be regenerated since that update was made.
sxa
commented
9 months ago Weekly pipeline with parameter releaseType=Weekly will trigger the regen https://ci.adoptium.net/job/build-scripts/job/weekly-openjdk11-pipeline/156/parameters/. The code is in https://github.com/adoptium/ci-jenkins-pipelines/pull/883/files.
The regen didn't happen as the typo. The fix PR adoptium/ci-jenkins-pipelines#907
@sophia-guo I'm seeing a lot of the testList_ jobs showing this warning (including hotspot ones) - should those be regenerated along with the main jobs, and if not can we force a regen of those? If not would it be a good idea to just delete them all since they will be regenerated oe would that cause problems?
Also Test_Job_Auto_Gen seems to have the issue for the TEST_JOB_NAME and LIGHT_WEIGHT_CHECKOUT parameters so that will require some manual intervention.
sophia-guo
commented
9 months ago There are two other types of test jobs naming as _testList_ and _rerun. Those jobs are generated if no jobs exist or if test job parameter GENERATE_JOBS is true, which currently can't be set by upstream weekly pipeline unless we add extra parameter.
sxa
commented
9 months ago if test job parameter GENERATE_JOBS is true, which currently can't be set by upstream weekly pipeline unless we add extra parameter
@sophia-guo So what option would you recommend? Sounds like there are three options:
And the third option of adding the extra parameter as an ongoing solution to the problem since it's quite likely this will happen in the future as parameters are changed so is a good idea. (We should create a separate issue for that in the pipelines repo)
sxa
commented
9 months ago From discussion with @smlambert in one of our calls today it is safe to delete all the testList jobs so I will start pruning ✂️
Edit: Since none were running I have now removed all TesttestList jobs from the jenkins server.
Removed jobs: old-testList-jobs.txt.gz) and less-old-testList-jobs.txt.gz
sxa
commented
9 months ago From the last day (checking after the above removal of testList jobs):
123 Test_openjdk17_hs_extended.functional_x86-64_linux
181 Test_openjdk17_hs_extended.openjdk_x86-64_linux
180 Test_openjdk17_hs_extended.perf_x86-64_linux
187 Test_openjdk17_hs_extended.system_x86-64_linux
110 Test_openjdk17_hs_sanity.functional_x86-64_linux
181 Test_openjdk17_hs_sanity.openjdk_x86-64_linux
180 Test_openjdk17_hs_sanity.perf_x86-64_linux
187 Test_openjdk17_hs_sanity.system_x86-64_linux
111 Test_openjdk17_hs_special.functional_x86-64_linuxI would have expected those to have been regenerated by the tagged builds, which I believe are running the equivalent of the "weekly" pipeline. Are we missing the part from the tagged pipelines that would perform this regeneration? Having said that, a lot of them seem to be for the PIPELINE_DISPLAY_NAME parameter - is that something that has been recently added?
smlambert
commented
9 months ago I am not aware that we have or support a PIPELINE_DISPLAY_NAME for those test jobs. Also, as noted in today's team call, I do not think that there has been a recent jdk17 triggered run, last one was 4 days ago.
sophia-guo
commented
9 months ago PIPELINE_DISPLAY_NAME was set for AQA_Test_Pipeline, which should not affect test jobs generated or triggered by non-AQA_Test_Pipeline. That is saying those warning message should not be in adoptium jenkins server log? It might be in in Eclipse temurin jenkins server log. As consume parameters dynamically in AQA Test Pipeline parameter PIPELINE_DISPLAY_NAME will pass to children test jobs, which will not fail tests but feel unnecessary. Might just not pass the parameter PIPELINE_DISPLAY_NAME to children jobs like some other parameters https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/aqaTestPipeline.groovy#L112-L114 @llxia
sophia-guo
commented
9 months ago The latest weekly jdk17 run was on Feb 4th https://ci.adoptium.net/job/build-scripts/job/weekly-openjdk17-pipeline/158/. The regen was set by weekly_release_pipeline.groovy. I don't think tagged builds will do the regen.
sxa
commented
9 months ago That is saying those warning message should not be in adoptium jenkins server log? It might be in in Eclipse temurin jenkins server log.
Just to clarify, I'm only looking at logs for ci.adoptium.net so everything I'm posting is related to that (Is that what you mean by "adoptium jenkins" as distinct from "Eclipse temurin jenkins" which is the Temurin Compliance one?
sxa
commented
9 months ago The latest weekly jdk17 run was on Feb 4th https://ci.adoptium.net/job/build-scripts/job/weekly-openjdk17-pipeline/158/. The regen was set by weekly_release_pipeline.groovy. I don't think tagged builds will do the regen.
OK then I think we need to look at setting it for the regular pipelines triggered by the beta triggers, since the weekly ones are irrelevant now with the tag-triggered jobs running all of the tests that we had in the weekly jobs.
sophia-guo
commented
9 months ago Yes, that's what confused me (PIPELINE_DISPLAY_NAME warning in ci.adoptium.net logs. )
sophia-guo
commented
9 months ago OK then I think we need to look at setting it for the regular pipelines triggered by the beta triggers, since the weekly ones are irrelevant now with the tag-triggered jobs running all of the tests that we had in the weekly jobs.
That might also mean alway regen the test jobs for pipeline - not sure is it necessary?
sxa
commented
9 months ago OK then I think we need to look at setting it for the regular pipelines triggered by the beta triggers,
That might also mean alway regen the test jobs for pipeline - not sure is it necessary?
I'm happy to take advice from you on any solution that resolves the warnings as that is the goal here :-) I was told previously that the regen was the primary way to prevent these from occuring when changes to the parameters occurred so that seemed like one suitable solution.
sxa
commented
9 months ago Noting that I triggered a reload of the configuration this morning since the logs were still showing issues in some of the testList jobs which were no longer there after yesterday's cleanup. I'll keep an eye on it for the next day and see which ones (if any) continue to show up.
llxia
commented
9 months ago PIPELINE_DISPLAY_NAME was added via https://github.com/adoptium/aqa-tests/pull/4186. It is for displaying info in the AQA Test Pipeline job only. It does not affect any child jobs, but it does not need to pass into child jobs. We should skip it.
llxia
commented
9 months ago Issue opened: https://github.com/adoptium/aqa-tests/issues/5089
sxa
commented
9 months ago sxa
commented
9 months ago @sophia-guo Is there a way to trigger regeneration of some of the jobs that aren't run regularly e.g. the external ones, and the non-hotspot ones like j9? Those seem to be most of the ones in the test area that are remaining.
sxa
commented
9 months ago The build ones that are left seem to be hotspot ones, or the pr tester (Mostly NODE_LABEL)
sophia-guo
commented
9 months ago @sophia-guo Is there a way to trigger regeneration of some of the jobs that aren't run regularly e.g. the external ones, and the non-hotspot ones like j9? Those seem to be most of the ones in the test area that are remaining.
Did you mean the not running jobs ( not triggered to run or disabled ) are still generating warning messages?
These aren't new but were discovered while working on #2108
There are a lot of warnings in the jenkins log which we should look at clearing up in the interests of avoid the risk of being unable to see the wood for the trees. All are related to
NODE_LABEL's use in various places.Presumably the parameters are being passed in from the upstream job somewhere but they are not defined on the callee so are superfluous. Can we wasily stop 'rogue' parameters being passed in or should we set the
-Dmentioned in the error?90 WARNING hudson.model.ParametersAction#filter: Skipped parameter
NODE_LABELas it is undefined onbuild-scripts/release/create_installer_mac. Set-Dhudson.model.ParametersAction.keepUndefinedParameters=trueto allow undefined parameters to be injected as environment variables or-Dhudson.model.ParametersAction.safeParameters=[comma-separated list]to whitelist specific parameter names, even though it represents a security breach or-Dhudson.model.ParametersAction.keepUndefinedParameters=falseto no longer show this message.