nthallen / monarch

Monarch Data Acquisition System
0 stars 1 forks source link

TMC Calibration Functions #109

Open nthallen opened 4 years ago

nthallen commented 4 years ago

When converting 32-bit data at high resolution, the generated conversions can overflow. This should at least be checked during the code generation phase (ensure that all intermediate values valid), and errors should be accompanied with some sort of helpful suggestion like "try specifying a lower resolution output format"