A Solid Foundation for Statistics in Python with SciPy

[mdhaber]

Overview of "A Solid Foundation for Statistics in Python with SciPy".

Expand tools for the analysis of variance

• [ ] multiway ANOVA - scipy/scipy#12630 (@WarrenWeckesser)
• [ ] nested ANOVA (@WarrenWeckesser)
• [ ] ANCOVA (@WarrenWeckesser)

New Statistical Tests

• [x] Kendall's tau-c - scipy/scipy#9361 (@mdhaber)
• [x] Page's L - scipy/scipy#12531 (@mdhaber)
• [x] Somers' D - scipy/scipy#12653 (@mdhaber)
• [x] Alexander-Govern - scipy/scipy#12873 (@DominicChm, @swallan)
• [x] Tukey-Kramer - scipy/scipy#13002 (@DominicChm, @swallan)

Improve Existing Tests

• [x] Confidence intervals - scipy/scipy#13371 (@mdhaber)

  • Binomial Test - scipy/scipy#12603 (@WarrenWeckesser)
  • pearsonr - scipy/scipy#12609

• [x] Options for one-sided p-values - scipy/scipy#12506, (@DominicChm)

  • ttests - scipy/scipy#12597
  • skewtest / kurtosistest / ranksums - scipy/scipy#13549
  • spearmanr/linregress - scipy/scipy#12801
  • mood - scipy/scipy#13008
  • ansari - scipy/scipy#13650
  • pearsonr - scipy/scipy#12609

• [x] Enhanced results for 2 x 2 contingency tables

  • Conditional maximum likeilhood odds ratio - scipy/scipy#13340 (@WarrenWeckesser)
  • Relative risk - scipy/scipy#13048 (@WarrenWeckesser)

Fitting Probability Distributions to Data

• [x] Add method of moments - scipy/scipy#11695 (@mdhaber)
• [x] Override generic fit methods where possible - scipy/scipy#11782 (@swallan)
  • laplace - scipy/scipy#11988
  • pareto - scipy/scipy#12457
  • rayleigh - scipy/scipy#12097
  • invgauss - scipy/scipy#12514
  • logistic - scipy/scipy#12738
  • gumbel - scipy/scipy#12737
  • powerlaw - scipy/scipy#13053
  • weibull
• [ ] Compute standard errors and confidence intervals of fit parameters (@WarrenWeckesser)
• [x] Enable fitting to censored data - scipy/scipy#13699 (@WarrenWeckesser)
• [x] Benchmark fitting - scipy/scipy#12639 (@swallan)

New Probability Distributions

• [x] negative hypergeometric distribution - scipy/scipy#11654, scipy/scipy#12170 (@mdhaber)
• [x] multivariate hypergeometric distribution - scipy/scipy#12585, scipy/scipy#12839 (@mdhaber)
• [x] noncentral hypergeometric distribution - scipy/scipy#13330 (@mdhaber)
• [x] Studentized Range Distribution - scipy/scipy#13732 (@DominicChm, @swallan)

Improve underlying code for PDF and CDF calculations

• [x] Boostinator - scipy/scipy#13328 (@mckib2)

Decrease Open Statistics Issues By the end of the project, we want the number of open stats issues to be below 282 (number of open stats issues on 3/18/2020), and preferably under 261 (number of open stats issues on 3/18/2020 created before project start date 2/1/2020). This is @mdhaber's list of issues to watch/fix; none need to be closed to finish the project, but it would be great to make a dent.

• [x] differential_entropy - scipy/scipy#13631
• [x] speedup hdquantiles - scipy/scipy#13566
• [x] quartile coefficient of dispersion - scipy/scipy#13475
• [x] JohnsonSU/SB doc - scipy/scipy#13444
• [x] Barnard exact test - scipy/scipy#13441
• [x] binomtest speedup - scipy/scipy#13433
• [x] Nakagami fitstart - scipy/scipy#13396
• [x] improved wilcoxon exact -scipy/scipy#13438
• [x] support returns wrong bounds with invalid parameters - scipy/scipy#13295
• [x] Cramer-von Mises 2samp - scipy/scipy#13263
• [x] mannwhitneyu overhaul - scipy/scipy#4933
• [x] vectorize mannwhitneyu - scipy/scipy#12837, scipy/scipy#11113
• [x] mannwhitneyu sparse arrays - scipy/scipy#11035
• [x] mannwhitneyu wrong? - scipy/scipy#9184
• [x] mannwhitneyu empty input - scipy/scipy#6897
• [x] mannwhitneyu nan input - scipy/scipy#4067
• [x] yet another mannwhitneyu issue - scipy/scipy#2118
• [x] some other mannwhitneyu issues - https://github.com/scipy/scipy/issues?q=is%3Aissue+is%3Aopen+mannwhitneyu
• [x] new cramervonmises test - scipy/scipy#11119
• [x] various pearson3 issues - scipy/scipy#12640
• [x] null hypothesis of KS test - scipy/scipy#12518
• [x] fix moment method for input arrays - scipy/scipy#12197
• [x] check for NaN in spearmanrho - scipy/scipy#12460
• [x] multivariate t distribution - scipy/scipy#12582
• [x] exact p-value in kendalltau - scipy/scipy#11563
• [x] gmeans weights - scipy/scipy#13067
• [x] asymmetric Laplace - scipy/scipy#12694
• [x] stats.binom.cdf issue - scipy/scipy#13079 (watching)
• [x] more binom - scipy/scipy#5122, scipy/scipy#7406, scipy/scipy#1603
• [x] permutation t-test - scipy/scipy#4824
• [x] scipy.stats.pearsonr/matplotlib weirdness - scipy/scipy#13035
• [x] multivariate distribution documentation - scipy/scipy#13103
• [x] what should stats tests return - scipy/scipy#13118
• [x] too many critical values from anderson_ksamp - scipy/scipy#11140
• [x] scipy.stats.skew roundoff error - scipy/scipy#11086 - @WarrenWeckesser will re-review/merge
• [x] stats.zscore roundoff error - scipy/scipy#12815
• [x] gausshyper distribution accepts invalid parameters - scipy/scipy#10134
• [x] rayleigh.fit issue - scipy/scipy#13071
• [x] Generalized Hyperbolic Distribution - scipy/scipy#13025
• [x] Masked arrays in binned_statistic_dd - scipy/scipy#12898
• [x] Documentation of stats.lognorm - scipy/scipy#12844
• [x] ttest_ind returns an object - scipy/scipy#12800
• [x] vectorize pearsonr - scipy/scipy#9307
• [x] scipy.special.bdtrik incorrect - scipy/scipy#11134
• [x] scipy.special.btdtri inaccurate - scipy/scipy#12794, scipy/scipy#12635
• [x] stats.ncx2 - scipy/scipy#11777
• [x] stats.ncx2 - scipy/scipy#12731
• [x] anderson-darling for Weibull - scipy/scipy#12787
• [x] weightedtau documentation mistake - scipy/scipy#12778
• [x] rv_histogram PR - scipy/scipy#12759 - @WarrenWeckesser will close
• [x] stats.pareto.fit raises RuntimeWarning - scipy/scipy#12545
• [x] spearmanr nan propagation - scipy/scipy#12411
• [x] Add nan_policy to stats.zmap - scipy/scipy#12403
• [x] loc is not noncentral - scipy/scipy#12102
• [x] chisquare should check normalization - scipy/scipy#12282
• [x] spearmanr constant input issues - scipy/scipy#12241
• [x] bugs in ks_2samp / ks_twosamp? - scipy/scipy#12212
• [x] tmean axis - scipy/scipy#12143, scipy/scipy#9770 - @WarrenWeckesser will reply
• [x] boxcox bounds - scipy/scipy#11911
• [x] getting just skewness slow - scipy/scipy#11742
• [x] zipfian distribution - scipy/scipy#11523
• [x] cdfchn fix - scipy/scipy#11477
• [ ] multivariate rvs shape - scipy/scipy#11150
• [x] stats docs examples - scipy/scipy#11148
• [x] loggamma instability - scipy/scipy#11094
• [x] user provided inverse covariance matrix - scipy/scipy#11053
• [x] mark slow tests - scipy/scipy#12132
• [x] constant input linregress - scipy/scipy#10896
• [x] ttest_ind default - scipy/scipy#10889
• [x] KDE bug? - scipy/scipy#10814 (Ask WW to double check at some point)
• [x] rv_continuous assumes scalar parameters - scipy/scipy#10661
• [x] Poisson 2samp E-test (stalled) - scipy/scipy#10561
• [x] Descriptive statistics nan_policy documentation - scipy/scipy#10519
• [x] gamma distribution fitstart problem? (No.) - scipy/scipy#10212 - @WarrenWeckesser will close.
• [x] reference for circstd - scipy/scipy#10096
• [x] kruskal nd input error - scipy/scipy#9981
• [x] cdf_distance with p=np.inf is KS - scipy/scipy#9916
• [x] Johnson SL distribution - scipy/scipy#9689
• [x] interval method description - scipy/scipy#9706
• [x] Yuen t-test - scipy/scipy#13696
• [x] method to list parameters of distributions - scipy/scipy#9575
• [x] skewed cauchy - scipy/scipy#7702
• [x] twp stats tutorials issues - scipy/scipy#7542 (PR in scipy/scipy#13233)
• [x] beta distribution pdf inf? - scipy/scipy#10713
• [x] frozen distribution support - scipy/scipy#13228
• [x] Yeo-Johnson LL function typo - scipy/scipy#13273
• [x] Issues with fitting several distributions - scipy/scipy#1884
• [x] boxcox w/ constant data - scipy/scipy#12225
• [x] scaled inverse chi-square - scipy/scipy#10933
• [x] ttest_ind_from_stats warning - scipy/scipy#10269
• [x] Kendall Tau distance - scipy/scipy#7205
• [x] wishart - scipy/scipy#6471, scipy/scipy#6472, scipy/scipy#6474
• [x] stats distribution/methods keyword collision - scipy/scipy#5982
• [ ] guess fit shape parameters - scipy/scipy#5362
• [x] inv->cho_solve - scipy/scipy#5087
• [x] ansari 'alternative' and 'exact' keywords - scipy/scipy#5012
• [x] stats distributions ppf-cdf roundtrip - scipy/scipy#3214
• [x] expect termination condition - scipy/scipy#2983
• [x] distribution._stats is not vectorized - scipy/scipy#2876
• [x] stats binom at non-integer n - scipy/scipy#2414
• [x] broken test in test_continuous_basic - scipy/scipy#2071

Outreach Event

Other

• [x] Boost Feature Comparison (@swallan)

[rlucas7]

@mdhaber you can checkoff the box for the 'check for NaN in spearmanrho' now

[mdhaber]

Yup, thanks!

[rlucas7]

@mdhaber you can checkoff box for the multivariate t distribution now too.

[WarrenWeckesser]

https://github.com/scipy/scipy/pull/11119 was merged, so you can check off the new cramervonmises test.

[WarrenWeckesser]

PR for the relative risk: https://github.com/scipy/scipy/pull/13048

[mdhaber]

@WarrenWeckesser Two more weeks left in the quarter...

[rlucas7]

@mdhaber you can check off the multivariate hypergeometric box:

 multivariate hypergeometric distribution - scipy#12585, scipy#12839 (@mdhaber)

[caos21]

I apologize if this is not the appropriate channel to open this discussion.

Following the issues covered in #11477, I would like to share my findings related to dcdflib, which is used to evaluate the cumulative density function (CDF).

• dcdflib was added to the SciPy's subversion repository on February 23, 2002 4423ed55, possibly from the SciPy's CVS repository, which I didn't found online.
• At least there are three versions of this library 1994, v1.1, and the version uploaded to scipy which is outdated by v1.1. The dcdflib v1.1 is available from netlib random here
• The same author of v1.1 offers a FORTRAN95 version with improvements called cdflib90
• A FORTRAN90 version is available here with all the functions in one file.

What can be done?

• I know that one possibility is to replace the existing dcdflib with the boost math toolkit.
• Another possibility is to update the actual version to v1.1.
• Use the F90 or F95 implementations.

In the last two cases, a follow-up of the code's modification has to be done.

Why?

• The actual code lacks precision and inline comments compared to the others.
• It could close a couple of issues.

I would be happy to help in any direction you decide.

[mdhaber]

Hi @caos21, thanks for mentioning this. @mckib2 is actually working on replacing parts of SciPy.stats with the Boost versions in #48. Would you be interested in taking a look at that? We're not going to change everything at once; this first PR will only actually replace SciPy's beta, binom, and nbinom distributions. The idea is to get all the machinery in place so that it will be easy to take things from Boost as needed in future PRs. Would this make it easy to replace cdflib with Boost's tools?

[caos21]

Hi and thank you @mdhaber , I think it is reasonable. But first, I would like to inspect how involved is cdflib in all SciPy. In the meantime, I can update cdflib to V1.1 and apply all the patches and modifications done in the past. In that way, I hope nothing breaks.

Should we move this discussion to #48 ?

[mdhaber]

In that case, it would probably be better to open an issue or PR on the main repo, or maybe email the mailing list to get wider attention. Only a few of us are working here now.

[caos21]

Perfect I will do, and after that, I will jump into boost #48 to see how can I be of use

[mckib2]

@mdhaber Don't know if we're interested in still keeping this list up to date:

• JohnsonSU/SB doc - scipy#13444 -- is merged
• Masked arrays in binned_statistic_dd - scipy#12898 -- is closed (not sure if resolved?)
• Proof of concept: wrap chi2 distribution from Boost - scipy#13328 (@mckib2) -- can be checked off or updated to scipy Boost PR?

[mdhaber]

Thanks @mckib2. We've been working from Monday.com recently, but it is still good to check these off.

[mdhaber]

Functions/distributions we might want to borrow from Boost:

• [x] ncf - gh-2877
• [x] ncx2 - gh-11777
• [ ] unregularized incomplete beta / logser - gh-3890
• [x] nct - gh-7104

[mdhaber]

@tupui At a glance, these are the issues and PRs we have open for multivariate distributions. Multivariate distributions represent ~1/5 of the number of open issues and PRs and issues with the scipy.stats label.

Multivariate Distributions - 30 of the 187 issues with scipy.stats label, 9 out of 52 PRs with scipy.stats label as of 3/14/2022.

Multivariate Normal - (Fewer than 150 lines of real code has all these issues and PRs.)

PRs

• [x] gh-12184
• [x] gh-11150
• [x] gh-9973
• [x] gh-9072
• [x] gh-5288

Issues

• [x] gh-15676
• [x] gh-15675
• [x] gh-15627
• [x] gh-15509
• [x] gh-15508
• [x] gh-13541
• [x] gh-11772
• [x] gh-11053
• [ ] gh-10328
• [x] gh-9942
• [x] gh-9921
• [x] gh-9257
• [x] gh-7496
• [x] gh-5286

New Distribution

PRs

• [x] gh-6739

Issues

• [ ] gh-14870
• [ ] gh-14325
• [x] gh-15618
• [ ] gh-7212
• [ ] gh-6528
• [ ] gh-6000

Other

PRs

• [x] gh-12312

Issues

• [x] gh-15677
• [ ] gh-15131
• [x] gh-14958
• [x] gh-13450
• [ ] gh-13014
• [x] gh-12799
• [x] gh-11860
• [ ] gh-11422
• [x] gh-7689
• [x] gh-6474
• [x] gh-6472
• [x] gh-6471
• [x] gh-6006

[mdhaber]

IIRC, several bugs involving constant input (i.e. all elements of a slice equal) have been reported. I'll collect them here as I run across them.

• [x] gh-15554
• [x] gh-14418
• [x] gh-13245
• [x] gh-11086
• [x] gh-10896

gh-13254 tried to address this for some functions, but I suspect it is a widespread problem.