We are taking measurements in millimetres, because that is about as good as the tools give us. Scientifically we should be estimating an extra significant digit.
However, because we want to avoid using floating point math, we should give ourselves an additional 2 significant digits.
So, our internal math will work with integer values in micrometers. This will help avoid rounding error in calculations. 32-bit integers give us more than enough precision for the size of track we are working with.
ferrous26
commented
10 years ago
We are taking measurements in millimetres, because that is about as good as the tools give us. Scientifically we should be estimating an extra significant digit.
However, because we want to avoid using floating point math, we should give ourselves an additional 2 significant digits.
So, our internal math will work with integer values in micrometers. This will help avoid rounding error in calculations. 32-bit integers give us more than enough precision for the size of track we are working with.