jenkinsci / benchmark-plugin

Benchmark plugin to compare boolean and numerical results between runs.
https://plugins.jenkins.io/benchmark/
MIT License
7 stars 6 forks source link

Add google benchmark schema #31

Open jsearles57 opened 5 years ago

jsearles57 commented 5 years ago

Add google benchmark schema. An example of what the schema might look like is at https://stackoverflow.com/a/53226336

tg-m commented 4 years ago

Yeah, I would also like to see this one working, since I cannot manage to create a custom schema for google benchmark. The referenced one seems not to work...

andrean-3lateral commented 1 year ago

I was able to use the reference benchmark schema from the stackoverflow post:

benchmark(
    altInputSchema: '''{
        "description": "Google Benchmark JSON schema",
        "failure": { "value": true },
        "type": "object",
        "properties": {
            "benchmarks": {
                "type": "array",
                "items": {
                    "type": "object",
                    "properties": {
                        "name": { "type": "name" },
                        "real_time": { "type": "result" },
                        "cpu_time": { "type": "result" },
                        "iterations": { "type": "parameter" }
                    }
                }
            }
        }
    }''',
    inputLocation: 'benchmark_results.json',
    schemaSelection: 'customSchema',
    thresholds: [[$class: 'PercentageThreshold', percentage: 1.0, testGroup: '', testName: 'cpu_time']],
    truncateStrings: false
)

this is the pipeline call, where I earlier executed google benchmark with the arguments to output results into a benchmark_results.json file.

The problem is that thresholds are not really usable, positive improvements are not allowed (e.g. faster execution in latest build compared to previous).