I was able to address the second 2 sources of numerical instability by adding
BooleanIndexing.replaceWhere(res, EPS, Conditions.lessThanOrEqual(EPS));
Right before the 2 places where sqrt is taken in the java mass method. This prevents the stdDev from becoming 0, and prevents a negative distance from becoming NaN in the final result. All results for distances and indices in my different manually added simple test cases work much better now. There are no more NaN's or Infinities in the results.
I was able to address the second 2 sources of numerical instability by adding BooleanIndexing.replaceWhere(res, EPS, Conditions.lessThanOrEqual(EPS)); Right before the 2 places where sqrt is taken in the java mass method. This prevents the stdDev from becoming 0, and prevents a negative distance from becoming NaN in the final result. All results for distances and indices in my different manually added simple test cases work much better now. There are no more NaN's or Infinities in the results.