search

byu-dml / d3m-dynamic-neural-architecture

1 stars 1 forks source link

Pearson and Spearman metrics return NAN #187

Open erikhuck opened 5 years ago

erikhuck commented 5 years ago

If the standard deviation of y or y_hat is zero, causing a divide by zero, we could add a small amount to the standard deviation that's zero, in order to avoid a divide by zero. This would mean that we would have to implement our own pearson and spearman metrics, but they would handle this edge case.

bjschoenfeld commented 5 years ago

For the constant models (i.e. mean model), there correlation is not defined, so NaN makes sense. Is there another case where these are metrics are returning NaN?

erikhuck commented 5 years ago

@bjschoenfeld A case that I found was particular datasets were returning NAN for their correlation, and then when the average and standard deviation across all datasets were computed, they too, of course, were NAN. This was even though the other datasets were not NAN. I actually discovered this doing the experiments with list-wise loss. I haven't seen it using the loss functions for this project, but it apparently is possible.

bjschoenfeld commented 5 years ago

Can you give specific examples and output?