magnetometer x y z calculations can overflow at less than half the sensor range

The CALIBRATED_SAMPLE calculation can overflow 32bit signed integer arithmetic.

#define CALIBRATED_SAMPLE(sample, axis) (((sample.axis - calibration.centre.axis) * calibration.scale.axis) >> 10)
#define LSM303_M_NORMALIZE_SAMPLE(x) (150 * (int)(x))

Raw sensor value range is up to 32767, so sampleENU is up to 4915050. calibration.scale.axis is around 1024, so the CALIBRATED_SAMPLE calculation can involve numbers greater than 5 billion, and overflow will occur just above 2 billion.

To see it happening, add trace as follows, and move a magnet close to the sensor.

https://github.com/lancaster-university/codal-core/blob/master/source/drivers/LSM303Magnetometer.cpp#L181 DMESGN( "raw %d enu %d ", (int) (*z), (int) sampleENU.z); https://github.com/lancaster-university/codal-core/blob/master/source/driver-models/Compass.cpp#L320 DMESG( "calibrated %d ", (int) sampleENU.z);

raw 13584 enu 2037600 calibrated 2037600
raw 14542 enu 2181300 calibrated -2013004
raw 13843 enu 2076450 calibrated 2076450

These figures are without calibration, but calibrated calibration.centre.xyz are small compared to these sampleENU.xyz values.

lancaster-university / codal-microbit-v2

magnetometer x y z calculations can overflow at less than half the sensor range #304