C computed values not matching up with calibration sheets (precision)

somts / odf-ctd-proc

ODF CTD processing c. 2016

BSD 3-Clause "New" or "Revised" License

8 stars 7 forks source link

C computed values not matching up with calibration sheets (precision) #9

Open asx- opened 7 years ago

asx- commented 7 years ago

Running sensor calibration sheet values through cond_dict() produces a result accurate to at least 3 decimal places, mostly 4. This falls short of the 5 decimal places given on the spec sheet when given in S/m.

webbpinner commented 7 years ago

I suspect this is related to the magnitude error. Recommend we solve that first.