keycloak / keycloak-benchmark

Keycloak Benchmark
https://www.keycloak.org/keycloak-benchmark/
Apache License 2.0
125 stars 68 forks source link

[Horreum] Infinispan metric calculation sometimes results to division to 0 #832 #833

Closed andyuk1986 closed 1 month ago

andyuk1986 commented 1 month ago

Fixes #832

andyuk1986 commented 1 month ago

You can find a successful run on my fork here: https://github.com/andyuk1986/keycloak-benchmark/actions/runs/9267087730

andyuk1986 commented 1 month ago

So, I am afraid that the generated report data so far was only correct for memoryUsageTest, for other 2 test runs (cpuUsage and cpuUsageForGrants was incorrectly calculated).

ahus1 commented 1 month ago

@andyuk1986 - thank you for figuring out the problem!

To help me understand the root cause: Can you please describe it to me? Looking at the diff, the rm seems to do the trick?

andyuk1986 commented 1 month ago

@andyuk1986 - thank you for figuring out the problem!

To help me understand the root cause: Can you please describe it to me? Looking at the diff, the rm seems to do the trick? @ahus1 actually there were 2 problems:

  1. The floating point math operation in bash sometimes were throwing error which was causing that data was completely missing for specific test. > I have fixed this by rounding the data already on prometheus level.
  2. The 'rm' part was the reason of devision to 0 issue. When the statistic is read from prometheus the data is stored in the file. This is done before and after the benchmark ran and is written on new line. Then the metric calculator reads the data and makes math operations to calculate that and theoretically should remove the file for next bechmark iteration as the next benchmark uses the same file name. I was deleting one of the files and missed the second, as for ispn metric calculation I am picking 2 time sub-metrics (seconds sum and count) and store them in different files for later processing. So as one of the files was not deleted, the later benchmarks were adding the new statistics to already existing file where already data was from previous benchmark run, and after that the whole calculation was going wrong.
ahus1 commented 1 month ago

Thank you for the explanation and the fix!