Potential fix for numerical instabilities for corr and partial_corr: z-normalize

[wolfgang-ganglberger]

Hi Raphael,

Big fan of your work and of this package, thanks so much. I was wondering if you have thoughts on numerical stability. The correlation and partial correlation can run into numerical instabilities when scales of variables are very different (factor 1e5 difference approximately). The functions do throw an error and the p-value is set to nan, however one fix could be to z-normalize the data before performing the matrix computations. Correlation and partial correlation functions are independent of scale.

See the screenshot for an example.

I think the pro is that it makes computations usually more robust, perhaps preventing people to use wrong results when numerical instabilities occur. A con might be it also does not solve all instabilities, e.g. if you have a variable with 0 standard deviation.

[raphaelvallat]

Hi @wolfgang-ganglberger,

Thanks for your kind words and the detailed proposal!

I think that sounds like a good approach to me, we just need to make sure not to overwrite the input data with the z-scored version of the variables.

Please feel free to submit a PR and we can continue the conversation there!

Thanks Raphael