Open piyush286 opened 5 years ago
@llxia @ShelleyLambert We are thinking of what might be the best option out of those mentioned above for using benchmark configs on Openjdk-tests Framework. Would you guys have some suggestions?
This design will take time to flesh out. As we have already discussed, we have identified that the config files in PerfNext could potentially be collapsed into the same format (with some additions) to the format as the playlist files.
I think it is important to distinguish between the 2 main use cases:
the nightly runs, which do not need to be dynamically generated, they are config files with default settings in them
the runs that a person using the PerfNext GUI to generate a config file, so overwriting default settings
Option 4 (essentially its your Option 2, but moving the script generation down to lower layer, as you no longer require it in Perfnext if you are calling tkg):
For 1) playlist file contains the test targets for the nightly build, that links to a default config file that is checked into the performance/subdir folder for a particular benchmark (in this way the nightlies do not depend at all on Perfnext instance to run, it is NOT acceptable to depend on Perfnext to be running, in order for tkg to run)
For 2), playlist file contains a test target called custom_benchmarkName, your are using Perfnext GUI, which generates a custom config that calls custom_benchmarkName target with a path or link to the location of the custom config file
Do not keep the logic for generating benchmark scripts (what Salman named BenchEngine) in Perfnext. It can be decoupled, moved to the lower layer. You avoid duplication, and simplify Perfnext, which essentially should be as dumb a GUI as possible. (as in, most if not all functionality should be possible without the GUI layer present).
We can discuss in a web conference soon, if this wasn't clear.
Some of the work in this issue might be related to Enable PerfNext To Use Openjdk-tests Framework.
Thanks Shelley for the suggestions.
Jotting down some more thoughts that came up so that I don't forget. We can discuss these thoughts as well in our meeting.
- the nightly runs, which do not need to be dynamically generated, they are config files with default settings in them
Totally agree!
Do not keep the logic for generating benchmark scripts (what Salman named BenchEngine) in Perfnext. It can be decoupled, moved to the lower layer. You avoid duplication, and simplify Perfnext, which essentially should be as dumb a GUI as possible. (as in, most if not all functionality should be possible without the GUI layer present).
/api/benchengine/submit
API currently sends 2 scripts to Jenkins Pipeline:
setupScript: It does all the setup tasks such as downloading benchmark and SDK packages.
benchmarkScript: Exports all the necessary environment variables and runs the main benchmark script.
We could get rid of the setupScript
since Openjdk-tests Framework can already do that. We would still need the ability to generate benchmarkScript
so that developers are able to change default configs and PerfNext can pass the custom benchmark script to Openjdk-tests Framework to run.
Yes, let's keep listing the requirements, and not get bogged down in how to accomplish it yet. It is important to clearly know "what needs to be accomplished" before getting distracted by "how it will be implemented".
Background of the Current Design for Running Benchmarks through Openjdk-tests Framework
In order to integrate all performance benchmarks into openjdk-tests repo, we need to add benchmark configurations for several benchmarks and variants that are able to run on all supported HW platforms by OpenJDK. As part of this initiative, we tested the water by deploying ODM benchmark on Openjdk-tests Framework since that benchmark is simple to configure and run, though it needs a license.
Initially, we designed the Openjdk-tests Framework to use a bash script (i.e.
benchmark.sh
as shown below) that would keep all the bash commands, which are required to set environment variables and run the benchmark.https://github.com/AdoptOpenJDK/openjdk-tests/blob/0cff000dabf262f48ad0c20f45f36892f0220fcb/performance/odm/scripts/benchmark.sh#L1-L55
Limitations of the Current Design
Some issues with keeping the benchmark configs as export commands inside a static bash script:
This design is not very flexible or maintainable if we want to run benchmarks with multiple variations.
Environment Variables are hard-coded for just one machine right now. So, you would need to manually change environment variables such as
CLIENT
,DB_MACHINE
,LIBERTY_HOST
andAFFINITY
while running some benchmarks, especially the ones that require more than one machine, such as Liberty DayTrader. You would also need to change the command for pinning threads such as numactl.Redundancy! Currently, all performance tests can be run from the PerfNext launcher (https://github.com/AdoptOpenJDK/openjdk-test-tools/tree/master/PerfNext/), which already has all the benchmark configurations in that repo.
If you see the ODM configs file from PerfNext below, you'll notice that those configs have been duplicated in the openjdk-tests repo for now. The only major difference is the format of configs. On openjdk-tests, we use a bash script that has all the configs in export commands, while on PerfNext, we keep the configs in a XML file.
https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b528a63ea146080f4bb1430740a734c2af87df7e/PerfNext/config/benchmarks/data_simple/ODM.xml#L1-L55
Possible Solutions
In order to avoid having the same benchmark configurations in multiple locations, we should use the configs from the PerfNext repo. Currently,
playlist.xml
(shown below) uses the static bash script (i.e.benchmark.sh
).playlist.xml
should call some tool/script to dynamically generate that bash script using the benchmark configs from PerfNext and then run that generated script.https://github.com/AdoptOpenJDK/openjdk-tests/blob/0cff000dabf262f48ad0c20f45f36892f0220fcb/performance/odm/playlist.xml#L18-L24
Option 1:
PerfNext also has an API (snippet below) that can generate the bash script for running a benchmark with a specific name and variant. It can also generate the specific HW environment variables such as CPU affinity commands and client machine for a selected machine. So we can possibly use that PerfNext's feature (
/api/benchengine/generatescript
) to generate the bash script for running benchmarks through Openjdk-tests Framework.Code snippet from https://github.com/AdoptOpenJDK/openjdk-test-tools/blob/b528a63ea146080f4bb1430740a734c2af87df7e/PerfNext/app/apis/BenchEngine/parser.js#L20-L34
Drawbacks of Option 1 This option would create an extra dependency on Openjdk-tests Framework since it would always require PerfNext to be up and running. If PerfNext is broken for some reason, then we wouldn't be able to run performance tests through Openjdk-tests Framework.
Option 2:
Drawbacks of Option 2 Redoing what PerfNext can already do. In future, if we decide to get the HW specific configs in some other way (i.e. get it directly from Jenkins labels instead of machine data XML file), then we would need to make changes to both PerfNext and translator script (XML to Shell).
Option 3:
Combined design of option 1 & option 2.
Assigned Contributors
Awsaf (@pinicman) from my team would work on adding this functionality.