Support server and client testing

[llxia]

In some cases, two machines are needed for testing. For example, some performance tests will need to communicate between server and client. We need to update our test pipeline to support this case.

Proposed solution:

• create a Jenkins job Machine-Reservation - this is a simple job to grep user-specified machine and sleep for the user-specified time. The main idea is to reserve the machine.
• in the test pipeline, a new parameter RELATED_NODES should be added. If the user provides a node name in RELATED_NODES, the test pipeline will trigger Machine-Reservation to book the specific machine before starting any testing. Otherwise, proceed with testing normally. For this case, we just need to pass in a single node. This can be further extended later for multiple nodes (i.e., comma separated).
• RELATED_NODES is set as an env variable to pass into the test scripts
• individual job will connect to RELATED_NODES for testing
• once the testing is completed, abort Machine-Reservation if it is still running

[llxia]

FYI @awsafsakif @smlambert

[awsafsakif]

We also need a check to see if CLIENT_NODE machine is idle before triggering the reservation job. Otherwise we will run into an issue where if the client machine is already busy running a different benchmark, the reservation job will be scheduled but not triggered.

Extra variables required: TIMEOUT: How long server machine will be active in jenkins till job is cancelled, INTERVAL: How often to check if CLIENT_NODE is idle before triggering machine reservation and continuing with the benchmark run.

[llxia]

Yes, for checking if a node is available https://stackoverflow.com/questions/37227562/how-to-know-whether-the-jenkins-build-executors-are-idle-or-not-in-jenkins-using/37238045

Also, we have similar code to use the API: https://github.com/adoptium/aqa-tests/blob/master/buildenv/jenkins/openjdk_tests#L300

[smlambert]

Created https://ci.adoptopenjdk.net/job/Machine-Reservation job as a placeholder for a pipeline described in this issue.

@norbline - if you have not already done so, please rebase your branch (described in Contributing.md) in preparation to begin work on this issue. I believe @llxia will share some code with you relating to Machine-Reservation shortly.

[llxia]

TODO:

• [x] create Jenkins job Machine-Reservation https://ci.adoptopenjdk.net/view/Test_grinder/job/Machine-Reservation/

• [x] create a separate Grinder for this work (Shelley)

• [x] checkin initial Machine-Reservation script in aqa-tests repo (Lan) - PR https://github.com/adoptium/aqa-tests/pull/3296

• [ ] For now, If RELATED_NODES is provided and TARGET=IdleMicrobenchmark_HS, trigger Machine-Reservation job with RELATED_NODES and TIME_LIMIT. The change should be made in the Test stage

• [ ] Update IdleMicrobenchmark_HS to echo RELATED_NODES https://github.com/adoptium/aqa-tests/blob/master/perf/idle_micro/playlist.xml#L17 (for testing purpose, do not need to check in)

• [ ] get Machine-Reservation job status (job number).

  • If the RELATED_NODES cannot be reserved within 30 mins, abort Machine-Reservation job and abort the job.
  • If the RELATED_NODES can be reserved within 30 mins, proceed to execute the test.

• [ ] in post-stage, release RELATED_NODES (abort Machine-Reservation job) and print the message

• [ ] Update readme with the use case for multiple machine testing

• [ ] Update testJobTemplate to generate RELATED_NODES

[smlambert]

Hi @norbline - can you add a comment in this issue, so I can assign it to you?

Grinder_Perforama job is created for you to use to help test your work: https://ci.adoptopenjdk.net/job/Grinder_Perforama/

When you "Build with parameters, you should see the new parameter RELATED_NODES at the bottom just above the Build button.

[norbline]

Thanks, Shelley. All good to get on

On Thu, Jan 27, 2022, 5:56 PM Shelley Lambert @.***> wrote:

Hi @norbline https://github.com/norbline - can you add a comment in this issue, so I can assign it to you?

Grinder_Perforama job is created for you to use to help test your work: https://ci.adoptopenjdk.net/job/Grinder_Perforama/

When you "Build with parameters, you should see the new parameter RELATED_NODES at the bottom just above the Build button.

— Reply to this email directly, view it on GitHub https://github.com/adoptium/aqa-tests/issues/2841#issuecomment-1023438077, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJOPRPBAM2GN5OEVUIJE6RDUYF2MJANCNFSM5DHIPHRA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

[smlambert]

[llxia]

@norbline For using Jenkins API to get job stage status, I found a similar thing at StackOverflow: https://stackoverflow.com/questions/37009906/access-stage-results-in-workflow-pipeline-plugin Could you give it a try both REST API and groovy code? Thanks

[norbline]

yes @llxia I will give it a try

[llxia]

After the investigation and experiments, we decided to use nested nodes instead of Jenkins API. In this way, we do not need to control the workflow/communication between two jobs. It is easy to maintain and less error-prone.

The key changes for nested notes are delivered via #3364 and #3379. Enhancements:

• [x] check if we can get RELATED_NODES in playlist.xml. If not, set RELATED_NODES as env variable (see https://github.com/adoptium/aqa-tests/issues/2841#issuecomment-1065478169)
• [x] add readme https://github.com/adoptium/aqa-tests/wiki/How-to-Run-a-Grinder-Build-on-Jenkins
• [x] (optional) add timestamp{} to provide more info https://github.com/adoptium/aqa-tests/issues/3396

https://github.com/adoptium/aqa-tests/blob/25c804602873d503ec596371875c33a49fa88ad2/buildenv/jenkins/openjdk_tests#L100-L236

[norbline]

Thanks @llxia

[llxia]

@awsafsakif could you test your case and let us know? Thanks.

[llxia]

Verified in Grinder that we can directly get RELATED_NODES in playlist. There is no need to set it as an env variable.

In the console output, test-docker-ubuntu2010-x64-2 is printed.

00:02:37.449  ===============================================
00:02:37.449  Running test IdleMicrobenchmark_HS_0 ...
00:02:37.449  ===============================================
00:02:37.449  IdleMicrobenchmark_HS_0 Start Time: Fri Mar 11 20:12:57 2022 Epoch Time (ms): 1647029577279
00:02:37.449  Nothing to be done for setup.
00:02:37.449  variation: -Xmx1024m
00:02:37.449  JVM_OPTIONS:  -Xmx1024m 
00:02:37.449  { itercnt=1; \
00:02:37.449  mkdir -p "/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../TKG/output_16470295765006/IdleMicrobenchmark_HS_0"; \
00:02:37.450  cd "/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../TKG/output_16470295765006/IdleMicrobenchmark_HS_0"; \
00:02:37.450  echo test-docker-ubuntu2010-x64-2; mkdir -p "/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../TKG/output_16470295765006/IdleMicrobenchmark_HS_0"; \
00:02:37.450  cd /home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../../jvmtest/perf/idle_micro; \
00:02:37.450  "/home/jenkins/workspace/Grinder_Perforama/openjdkbinary/j2sdk-image/bin/java"  -Xmx1024m  \
00:02:37.450  -cp ":/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../../jvmtest/perf/idle_micro/IdleMicrobenchmark.jar" net.adoptopenjdk.test.gc.idlemicrobenchmark.IdleMicroBenchmark \
00:02:37.450  --memoryLimit=700 --time=60 --idletime=300 --warmup=1 >"/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../TKG/output_16470295765006/IdleMicrobenchmark_HS_0"/std.out 2>&1 ; \
00:02:37.450  if [ $? -eq 0 ] ; then echo ""; echo "IdleMicrobenchmark_HS_0""_PASSED"; echo ""; cd /home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/..;  else echo ""; echo "IdleMicrobenchmark_HS_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/home/jenkins/workspace/Grinder_Perforama/aqa-tests/TKG/../TKG/output_16470295765006/TestTargetResult";
00:02:37.450  test-docker-ubuntu2010-x64-2

FYI @awsafsakif

[llxia]

The feature is implemented. The task about - add timestamp{} to provide more info is optional and it is tracked in https://github.com/adoptium/aqa-tests/issues/3396 Close this issue.