Open madrjor02-bh opened 2 months ago
The bug does seem to exist in df.corr
The reason is the Welford's method which is used for calculation which doesn't perform well when the datasets has extremely small differences or precision issues.
import pandas as pd
import numpy as np
import math
values = [{'col1': 30.0, 'col2': 116.80000305175781},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None},
{'col1': 30.100000381469727, 'col2': 116.8000030517578},
{'col1': None, 'col2': None},
{'col1': None, 'col2': None}]
data = pd.DataFrame(values)
def corr_coef(X,Y):
x1 = np.array(X)
y1 = np.array(Y)
x_m=x1.mean()
y_m=y1.mean()
numer=0
v1sq=0
v2sq=0
for i in range(len(x1)):
xx = (x1[i]-x_m)
yy = (y1[i]-y_m)
numer+=xx*yy
v1sq+=xx*xx
v2sq+=yy*yy
return(numer/(math.sqrt(v1sq*v2sq)))
data = data.dropna()
corr_coef(data.iloc[:,0],data.iloc[:,1])
-0.7071067811865475
We get the same -0.707.. results if np.corrcoef() is used.
Though this is a rare case to happen in reality, I believe it should be fixed. I can work of fixing the issue by changing the Welford's method to the standard method for df.corr. It might slightly impact the time for running. Not sure if it will be acceptable by the team.
Thanks for the report!
It might slightly impact the time for running.
If it is slight, I think the numerical stability would be valued.
take
Pandas version checks
[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the latest version of pandas.
[X] I have confirmed this bug exists on the main branch of pandas.
Reproducible Example
Issue Description
In the code snipped I'm trying to calculate the correlation between a pair of columns. However, when using pearson correlation method for this particular example, the outputted correlation is outside the -1 to 1 expected range.
Expected Behavior
The output of the pearson correlation method should be inside the -1 to 1 range.
Installed Versions