search

dtcenter / MET

Model Evaluation Tools
https://dtcenter.org/community-code/model-evaluation-tools-met
Apache License 2.0
78 stars 24 forks source link

Improve logic in library for checking equality of two floating point numbers. #37

Open dwfncar opened 13 years ago

dwfncar commented 13 years ago

The is_eq function is defined In the file:
  svn-met-dev.cgd.ucar.edu:trunk/metsrc/basic/vx_math/is_bad_data.h
Currently, is_eq checks for equality between two doubles by checking to see if their difference is less than a specified tolerance value (default = 10E-10). Setting the tolerance value has proven to be a tricky issue since some data values are very small (like specific humidity) and some data values are large (like pressure). The task is to improve this logic for checking equality between two numbers by checking a certain number of significant digits - rather than a fixed tolerance value.

This task should not be done until automated testing is in place to ensure that the fix doesn't change the output statistics. [MET-37] created by johnhg

dwfncar commented 12 years ago

Review status in release 4.0. Should we be looking at relevant tolerance? by bonny