Percentile and Median : Document algorithms used

apache / jmeter

Apache JMeter open-source load testing tool for analyzing and measuring the performance of a variety of services

https://jmeter.apache.org/

Apache License 2.0

8.35k stars 2.1k forks source link

Percentile and Median : Document algorithms used #4399

Open asfimport opened 7 years ago

asfimport commented 7 years ago

@pmouawad (Bug 61130): Issue related to https://github.com/apache/jmeter/issues/4385 and https://github.com/apache/jmeter/issues/4388

OS: All

Duplicates:

asfimport commented 7 years ago

Aleksei Balan (migrated from Bugzilla): Hello Apache community.

I use JMeter almost every day and attach its data to my reports. That is why I care about accurate results so much. The median is calculating incorrectly. It is not corresponding to wikipedia example. https://en.wikipedia.org/wiki/Median (example at the top right corner of the page) The percentile is calculating incorrectly. It is not corresponding to wikipedia example. https://en.wikipedia.org/wiki/Percentile (Example 3) To help with the fixes I provided pull requests for both of them.

And now you are discussing is it a bug or a feature and wondering should something be changed or not.

All right then.

asfimport commented 7 years ago

@pmouawad (migrated from Bugzilla): Hello Aleksei, I didn't want to be rude by closing issue. First thanks for your time, your PR and work.

I closed the PR for the following reasons:

My understanding is that on large set of data which should be the case for correctly ran tests, the different percentiles algorithm will lead to similar values. "There is no standard definition of percentile,[2][3][4] however all definitions yield similar results when the number of observations is very large and the probability distribution is continuous"
As per Felix note, there doesn't seem to be 1 truth
Changing the algorithm might lead to changes in results that will trigger potentially a lot of "false" issues.

Still maybe there is a better way of fixing the issue, SO I reopened the bugs. It appears to me we could fix median as per your PR and use Math Percentile setup with R_1 accross all JMeter library.

But I won't touch them myself as I have no firm opinion.

Thank you

asfimport commented 7 years ago

Aleksei Balan (migrated from Bugzilla): Hello Apache community,

Can you please review these two pull requests https://github.com/apache/jmeter/pull/295 https://github.com/apache/jmeter/pull/296

They are fixing some calculation defects. I understand that these errors affect the results very rarely (especially on big number of samples) but still I think it is a good idea to have more precise calculations.

Thank you.

Aleksei Balan