Order of magnitude error in python pressure conversion

ControlEverythingCommunity / MS5803-14BA

Altimeter and Temperature Sensor

2 stars 0 forks source link

Order of magnitude error in python pressure conversion #1

Open NickCrews opened 5 years ago

NickCrews commented 5 years ago

Hi! There is an error in the calculation of the pressure in the python examples, where you should divide by 10 and not by 100:

error 1 error 2

This can be verified by the final step in the conversion process described in figure 15 of the datasheet.

Can you fix this, or should I submit a pull request?

NickCrews commented 5 years ago

There is also another error just above the last, where OFF2 is calculated by dividing by 8, when it should be divided by 2. Finally, by not using integer division in the calculations there is some variation between what the example provided in the datasheet and what the library calculates.