Unidata / ncWMS

Snapshots of ncWMS used by TDS
8 stars 7 forks source link

Fix handling of missing values #16

Closed joanpau closed 8 years ago

joanpau commented 8 years ago

Handling of missing values in DataChunk.readFloatValue is broken due to floating precission issues. For the missing and scale conversion the actual data types are completely ignored, and the fill value is internally read as double while the data value is read as float, possibly resulting in a different value. The later is preserved by the promotion to double and causes the comparison to fail. This is an example of this situation:

int i = -2147483647;
float f = (float) i;
double d = (double) i;
double f2d = (double) f;
System.out.format("(i) %d = (f) %.5f = (d) %.5f = (f2d) %.5f\n ", i, f, d, f2d);
// Outputs:
// (i) -2147483647 = (f) -2147483648.00000 = (d) -2147483647.00000 = (f2d) -2147483648.00000

Fix it by reading the data value as double, delaying the cast to float to the return statement.