search

jenkinsci / peass-ci-plugin

Jenkins plugin for peass to support performance measurement in CI
https://plugins.jenkins.io/peass-ci/
GNU Affero General Public License v3.0
4 stars 13 forks source link

No correct statistics-configuration for measurement overview #65

Closed mawHBT closed 2 years ago

mawHBT commented 2 years ago

If I specify a significance of 0.01, changes with t value between 5 and 10 for 2 VM executions are not found. It seems like Peass internally uses a different significance level.

DaGeRe commented 2 years ago

Thanks for the hint, this should be fixed in https://github.com/DaGeRe/peass/commit/3d2505778a178d0b376d90325625e8de60eb3350, could you check this?

DaGeRe commented 2 years ago

The reason for the still sometimes surprising behavior is, that commons-math implementation of heteroscedatic t-test was used, which normalizes variances. Especially for small sample sizes and differing variances, this leads to big differences in the p-value of the t-test and the t-value.

Therefore, since https://github.com/DaGeRe/peass/commit/acdca0f078672865bbe5fae6f43e45de90a7c4f4, a homoscedatic t-test is used. Does this work for you?

mawHBT commented 2 years ago

This works now.