Ensure that bench downloading script downloads artifacts from correct job

enso-org / enso

Hybrid visual and textual functional programming.

https://enso.org

Apache License 2.0

7.31k stars 317 forks source link

Ensure that bench downloading script downloads artifacts from correct job #9355

Open Akirathan opened 3 months ago

Akirathan commented 3 months ago

A follow-up of #9322. The Benchmark Engine workflow now has two jobs. One that runs with Oracle GraalVM and another one that runs with GraalVM CE. Ensure that the website_regen.py fetches artifacts from the GraalVM CE job so far.

JaroslavTulach commented 3 months ago

Ensure that ... fetches artifacts from the GraalVM CE job so far.

It'd be even better if the official benchmark results displayed values from both GraalVM versions. Each in different color.

At the end the goal is to find out why using Oracle GraalVM is better than sticking with GraalVM CE. Performance is likely one of the reasons. Having such reason directly that visible in the benchmarks result might be the convincing point.

Akirathan commented 3 months ago

First successful engine-benchmark.yml workflow run is at https://github.com/enso-org/enso/actions/runs/8257237123:

The uploaded Runtime Benchmark Report artifact contains results from the Engine Benchmarks (GraalVM CE). There are no artifacts for the Oracle GraalVM job.

GitHub
Benchmark Engine · enso-org/enso@ef82637
Hybrid visual and textual functional programming. Contribute to enso-org/enso development by creating an account on GitHub.

Akirathan commented 3 months ago

Preliminary manual comparison of GraalVM CE VS Oracle GraalVM engine benchmarks:

Label	GraalVM CE	Oracle GraalVM
org.enso.compiler.benchmarks.module.ImportStandardLibrariesBenchmark.importStandardLibraries	159.612	94.404
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy	0.120	0.090
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverDelegatingProxy	0.120	0.089
org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverVector	0.081	0.033
org.enso.interpreter.bench.benchmarks.semantic.EqualsBenchmarks.equalsTrees	1.002	0.614
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.caseBench3	2.800	2.201
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.caseBench6	3.205	0.989
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench3	2.566	0.446
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench6	2.980	6.048
org.enso.interpreter.bench.benchmarks.semantic.IfVsCaseBenchmarks.ifBench6In	2.951	12.164

JaroslavTulach commented 3 months ago

Re. ifBench6 - the manual inlining written by us is used in this benchmark - possibly it doesn't work well with Oracle GraalVM. Workaround: disable it and let Oracle do its job!

Otherwise the improvements look very nice!

Akirathan commented 3 months ago

Re. ifBench6 - the manual inlining written by us is used in this benchmark - possibly it doesn't work well with Oracle GraalVM. Workaround: disable it and let Oracle do its job!

Related to the work done in https://github.com/enso-org/enso/issues/5709

enso-bot[bot] commented 2 weeks ago

Pavel Marek reports a new STANDUP for today (2024-06-10):

Progress: - Making sure that results from Oracle GraalVM benchmarks are persistent.

bench result reporter in benchmarks-common refactored to report a single JSON file, according to a specific schema.
Most of the bench_download.py tool will get deprecated soon, let's replace it with a typescript CLI project.
- Much easier to maintain, test, etc.
- There is official octokit library for the GH API, unlike to the Python one.
- Struggling with even generating venv for Python
- This will be a HUGE simplification - just upload data from a single Github Action, this should be just a few dozens of JS code. It should be finished by 2024-06-12.

enso-bot[bot] commented 2 weeks ago

Pavel Marek reports a new STANDUP for today (2024-06-11):

Progress: - Closing stale PRs

Adding more tests for BindingsMap.exportedSymbols.
Implementing new tool for downloading benchmarking data and uploading them to engine-bench-results.
- Is a breeze with ockokit and NodeJS. It should be finished by 2024-06-12.

Akirathan commented 2 weeks ago

Add https://github.com/enso-org/enso/issues/9355#issuecomment-1993802484 : No artifacts for Oracle GraalVM benchmark job means that we have to gather the data from the job output. Note that job outputs also expire after 3 months, like artifacts.

Akirathan commented 2 weeks ago

Last 3 months of Oracle GraalVM bench results persisted in https://github.com/enso-org/enso/pull/10224/commits/70cb785c869081605ed11e8b2793896c4926f1d4

enso-bot[bot] commented 2 weeks ago

Pavel Marek reports a new STANDUP for today (2024-06-12):

Progress: - Results from Oracle GraalVM benchmark runs gathered in https://github.com/enso-org/enso/pull/10224#issuecomment-2163471412

It was more work than expected, as I needed to parse the results from job outputs - there are no artifacts for EE benches. It should be finished by 2024-06-12.