search

ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
305 stars 22 forks source link

Can engineers force memory reporting into one consistent scale unit (K or M)? #237

Open vanvoorden opened 5 months ago

vanvoorden commented 5 months ago

Hi! I just started using the package and I like it a lot! I am running two parallel benchmarks with grouping metric to compare the memory usage between the two. I am seeing results like this:

Memory (resident peak)
╒════════════════════════════════════════════════════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╤═════════╕
│ Test                                                   │      p0 │     p25 │     p50 │     p75 │     p90 │     p99 │    p100 │ Samples │
╞════════════════════════════════════════════════════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╪═════════╡
│ First (K)                                              │    8913 │   10273 │   10273 │   10273 │   10273 │   10273 │   10273 │     100 │
├────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ Second (M)                                             │      10 │      10 │      10 │      10 │      10 │      10 │      10 │     100 │
╘════════════════════════════════════════════════════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╧═════════╛

It's a little weird because one metric is reporting as 10273K and the other metric is reporting as 10M… do I have any option to pass a flag (similar to timeUnits) that can force that reporting to one unit so I can compare them a little easier? Thanks!

hassila commented 5 months ago

Hmm, I seem to remember we wanted to force the units to be the same and actually had an option for that / but couldn't see it now.

Would make sense to have the option - in the meantime you can try exploring some of the other output options here: https://swiftpackageindex.com/ordo-one/package-benchmark/1.22.4/documentation/benchmark/exportingbenchmarks

I seem to remember that you could compare two runs in jmh?

Away for a couple of weeks now and don't have access to a machine at the moment - so apologies for a bit fuzzy answers.

vanvoorden commented 5 months ago

@hassila Thanks!