KS p-values are different than those of other softwares

[anilaba]

I've been triple checking MLE alpha estimates and KS p-values with EViews, EasyFit and RealStatistics (Excel add-in). I get the same scaling parameter results, however the KS test results are different. The above three give the same result together, only yours is different. Which makes me think that it is likely that there is something wrong, maybe, with your code...

I am new here, this is my first ever post. Signed up just to point this out. Excuse all my mistakes, if there is any, regarding community behavior.

[csgillespie]

Thanks for the issue. This is definitely the right place to raise an issue.

• Can you provide a very simple data set (a few numbers) and give me results from the other software?
• My gut feeling is that I'm doing it "correct", the others are wrong. Normally I would doubt my code over three other pieces of software (that would be the safe bet), but this package fits powerlaws in a relatively technical way based on the Clauset 2009 paper. Look at http://www.real-statistics.com/, they don't mention powerlaws, which implies they are using the standard, but in correct, method. Of course, I could be wrong! The trouble with powerlaws is the very long tails

[anilaba]

Attached is the data with 106 observations. Your code yields x_min = 690 alpha = 3,194299 n_tail = 79 KS p-value = 0,1432

Real Statistics do not bootstrap for an x_min, he just runs an MLE for the given data set, as it is. However, after finding that the optimum count is 79 (via your code), I run his MLE add-in with top 79 observations, to compare. And I get an alpha of 2,194299!!! This is exactly "1" less than yours, even the decimals are exactly the same. I guess the reason is that his underlying Pareto PDF is different than yours (there is a +1 in yours), so that for all my comparisons, Real Statistics, EViews and EasyFit give the exact result for alpha ,-1 scaled [ your alpha = their alpha + 1]. The methods/codes runs the same MLE, only difference seems to be the underlying definitions of PDF & CDF. (Some people, in the literature, have said that there is underestimation due to different methods; but I believe it is due to the definition of the PDF).

Then, for the attached data, Real Stats add in, gives a KS_stat = 0,0759 , a critical value 0,1527 ( = 1,3581 / 79^0,5), and a corresponding KS p-value of 1,00. A weird value. But still, which means we cannot reject the null, "accepting" that we have a pareto fit, since the maximum difference between distributions is very small (a small KS stat). However, your code yields a KS p-value 0,1432 , which would reject at 0.15-0.20 level.

Whereas EViews yields (n=79, Pareto EDF test, MLE) same (scaled) alpha with a KS p-value of 0,0000. Another weird number. (In my version Eviews 8 reports KS stat if entered alpha and min as fixed parameters).

Hopefully I'm not totally off and wasting your time 🤔

Thanks in advance, anyway...

github_KS_powerlaw.xlsx

[Lvulis]

This is quite late and maybe too late for you, but for the record Colin Gillespie's KS is implemented 100% correctly. Some notes that may be of use:

(1) Per Clauset's 2009 paper and actually going back to the original English-language papers for the the KS test, when you fit parameters of a parametric distribution G to an empirical distribution F and want to test if the empirical distribution is not significantly different from that parametric distribution, the "critical value" tables do not hold. You must do a bootstrap exercise where you simulate the experiment fitting the data a large number of times, which is what Clauset et al., (2009) detailed. This is exactly what's done and is the right way to do it. This is a contemporaneous paper that documented the same issues with the test: https://doi.org/10.1175/MWR3326.1

For your data: (2) You should be wary of correctly using the discrete versus continuous estimators. The discrete estimator here means integer valued data, not just that it is it a discrete PDF in a formal sense. If your data corresponds to the counts of objects, you should use the discrete estimators (and therefore classes), it converges closer to the true parameter values and the KS test is done "correctly". Note using the discrete you will likely have found a lower (more significant) KS value, so it doesn't seem you've used the discrete MLE in poweRlaw in this case, otherwise part of the discrepancy between the different software packages you're using could be due to this.

For your interpretation: (3) Clauset et al. (2009) have a very important section detailing that you MUST test whether the power law is statistically a better fit than another long-tailed PDF, and the power of the test depends on the number of samples. You can observe in their assessment of the test that below 2,000 samples it's essentially impossible to distinguish a PL from an LN, so for your ~100 sample data unless you have a very strong reason to believe its power law, you may not wanna go through with this exercise.