search

LachlanGunn / oic

Open Instrument Control
MIT License
50 stars 18 forks source link

Many digit floating point numbers break scpi_parse_numeric() #8

Open palmerr23 opened 2 years ago

palmerr23 commented 2 years ago

When passed numeric arguments with more than ten significant digits, exponent_multiplier overflows. The mantissa is then multiplied by 1073441896 instead of the correct multiplier.

Changing exponent_multiplier from long to float fixes the issue, with only a trivial loss of precision.

LachlanGunn commented 2 years ago

Thanks for this. I'm a bit wary of switching to float, but then again I guess it is uncommon to need more than ten digits of precision. I'm a bit snowed under at the moment, but will have a look when I can to see if there is another solution that doesn't require floating-point, otherwise switch exponent_multiplier to float as you suggest.

palmerr23 commented 2 years ago

Thanks for the respoinse.

I didn't choose 10 digits! I was using TestController to generate automated test values and it produced the 10 digit numbers. I found a function in TC to trim the number of digits, so all is well.