enso-org / enso

Hybrid visual and textual functional programming.
https://enso.org
Apache License 2.0
7.34k stars 323 forks source link

Gather all data from benchmarks #10224

Open Akirathan opened 3 months ago

Akirathan commented 3 months ago

Closes #9355

Pull Request Description

Gather benchmarking data in a new, JSON format. Previously, we gathered only one score value per result. Now, everything that is provided by JMH framework is gathered.

The produced bench-results.json can be simply read locally. An example from few dummy runs is:

{
  "schema" : "results_schema.json",
  "results" : [ {
    "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverVector",
    "timestamp" : "2024-06-10T12:33:54.026187718",
    "score" : 0.0018004264683645476,
    "scoreError" : "NaN",
    "samples" : 1,
    "min" : 0.0018004264683645476,
    "max" : 0.0018004264683645476,
    "mean" : 0.0018004264683645476,
    "stdDev" : "NaN",
    "warmupIterations" : 1,
    "warmupMillis" : 1000,
    "measureIterations" : 1,
    "measureMillis" : 1000,
    "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002",
    "branch" : "wip/akirathan/9355-bench-res-json",
    "configuration" : {
      "osName" : "Linux",
      "osArch" : "amd64",
      "osVersion" : "6.5.0-35-generic",
      "vmName" : "OpenJDK 64-Bit Server VM",
      "vmVersion" : "21.0.2+13-jvmci-23.1-b30",
      "vmVendor" : "GraalVM Community",
      "jdkVersion" : "21.0.2"
    }
  }, {
    "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArray",
    "timestamp" : "2024-06-10T12:34:44.508761575",
    "score" : 0.0016180807887320298,
    "scoreError" : "NaN",
    "samples" : 1,
    "min" : 0.0016180807887320298,
    "max" : 0.0016180807887320298,
    "mean" : 0.0016180807887320298,
    "stdDev" : "NaN",
    "warmupIterations" : 1,
    "warmupMillis" : 1000,
    "measureIterations" : 1,
    "measureMillis" : 1000,
    "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002",
    "branch" : "wip/akirathan/9355-bench-res-json",
    "configuration" : {
      "osName" : "Linux",
      "osArch" : "amd64",
      "osVersion" : "6.5.0-35-generic",
      "vmName" : "OpenJDK 64-Bit Server VM",
      "vmVersion" : "21.0.2+13-jvmci-23.1-b30",
      "vmVendor" : "GraalVM Community",
      "jdkVersion" : "21.0.2"
    }
  }, {
    "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArrayProxy",
    "timestamp" : "2024-06-10T12:34:44.53703994",
    "score" : 0.002464619691933143,
    "scoreError" : "NaN",
    "samples" : 1,
    "min" : 0.002464619691933143,
    "max" : 0.002464619691933143,
    "mean" : 0.002464619691933143,
    "stdDev" : "NaN",
    "warmupIterations" : 1,
    "warmupMillis" : 1000,
    "measureIterations" : 1,
    "measureMillis" : 1000,
    "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002",
    "branch" : "wip/akirathan/9355-bench-res-json",
    "configuration" : {
      "osName" : "Linux",
      "osArch" : "amd64",
      "osVersion" : "6.5.0-35-generic",
      "vmName" : "OpenJDK 64-Bit Server VM",
      "vmVersion" : "21.0.2+13-jvmci-23.1-b30",
      "vmVendor" : "GraalVM Community",
      "jdkVersion" : "21.0.2"
    }
  }, {
    "label" : "org.enso.interpreter.bench.benchmarks.semantic.VectorBenchmarks.averageOverArrayProxyNew",
    "timestamp" : "2024-06-10T12:34:44.546709051",
    "score" : 0.0020222642668386607,
    "scoreError" : "NaN",
    "samples" : 1,
    "min" : 0.0020222642668386607,
    "max" : 0.0020222642668386607,
    "mean" : 0.0020222642668386607,
    "stdDev" : "NaN",
    "warmupIterations" : 1,
    "warmupMillis" : 1000,
    "measureIterations" : 1,
    "measureMillis" : 1000,
    "commitId" : "e3322694913312d01f70c8af6b8b06ed78cfa002",
    "branch" : "wip/akirathan/9355-bench-res-json",
    "configuration" : {
      "osName" : "Linux",
      "osArch" : "amd64",
      "osVersion" : "6.5.0-35-generic",
      "vmName" : "OpenJDK 64-Bit Server VM",
      "vmVersion" : "21.0.2+13-jvmci-23.1-b30",
      "vmVendor" : "GraalVM Community",
      "jdkVersion" : "21.0.2"
    }
  } ]
}

Important Notes

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

Akirathan commented 3 months ago

Since there are no artifacts for Oracle GraalVM benchmark jobs, I had to parse the data from job outputs. https://github.com/enso-org/enso/pull/10224/commits/70cb785c869081605ed11e8b2793896c4926f1d4 adds all these results that were gathered via npm run start in the benchmark-data project. Let's keep those data in this PR for now.