Loss of precision in downloaded data

[dbbotkin]

When measuring micro-Amp data, the 289 display shows: 1097.1 microA, for example. Data downloaded to Fluke FormView confirms 1097.1, however data printout from QSRR shows 0.0011 A for all samples. That would be a correct value with rounding, however that loses 3 digits of precision.

This might result from the way I'm using the script (long story) so I'll keep working on it.

[dbbotkin]

From 'get_double': return round(struct.unpack('!d', endian)[0],5) round to 5 places?? Not working for me . . . . This needs a fix I think

[dbbotkin]

My solution was to add definition: def get_double_value(string, offset): endian_l = string[offset+3:offset-1:-1] if offset > 0 else string[offset+3::-1] endian_h = string[offset+7:offset+3:-1] endian = endian_l + endian_h return round(struct.unpack('!d', endian)[0],8)

to use with values where precision was important. The get_double was for date times with "round 5"

works for me

[N0ury]

Thanks for reporting. I'll do some checks to see if there are no side effects. If ok I'll replace the "5" with "8". I must admit that for µA I use an external board because the 287 is not responsive enough for frequent changes.

[N0ury]

@dbbotkin I have tried some measurements in µA. Here are the results:

Round 5
Primary	Minimum	Maximum
5e-05 ADC	5e-05 ADC	5e-05 ADC
5e-05 ADC	5e-05 ADC	5e-05 ADC
5e-05 ADC	5e-05 ADC	5e-05 ADC
5e-05 ADC	5e-05 ADC	5e-05 ADC
5e-05 ADC	5e-05 ADC	5e-05 ADC

Round 8
Primary	Minimum	Maximum
5.28e-05 ADC	5.28e-05 ADC	5.281e-05 ADC
5.282e-05 ADC	5.282e-05 ADC	5.284e-05 ADC
5.284e-05 ADC	5.284e-05 ADC	5.286e-05 ADC
5.286e-05 ADC	5.286e-05 ADC	5.288e-05 ADC
5.288e-05 ADC	5.288e-05 ADC	5.29e-05 ADC

FlukeView part 1
Primary	Minimum	Maximum
52,80 µA DC	52,80 µA DC	52,81 µA DC
52,82 µA DC	52,82 µA DC	52,84 µA DC
52,84 µA DC	52,84 µA DC	52,86 µA DC
52,86 µA DC	52,86 µA DC	52,88 µA DC
52,88 µA DC	52,88 µA DC	52,90 µA DC

FlukeView part 2
Primary	Minimum	Maximum
5.28e-005 A DC	5.28e-005 A DC	5.281e-005 A DC
5.282e-005 A DC	5.282e-005 A DC	5.284e-005 A DC
5.284e-005 A DC	5.284e-005 A DC	5.286e-005 A DC
5.286e-005 A DC	5.286e-005 A DC	5.288e-005 A DC
5.288e-005 A DC	5.288e-005 A DC	5.29e-005 A DC

I would like to have your opinion before implementing this modification. For the FlukeView data, it is a manually reworked csv export. I haven't seen any side effects, but more tests should be necessary.

[dbbotkin]

Excellent! I'll get back to you in about an hour . . .

[dbbotkin]

There is documentation available on analog to digital conversion "ADC" chips ability to manage precision, accuracy, speed and repeatability--I can send you links if you'd like--however the limiting factor is the number of bits the ADC can convert. The more bits, the more digits available for display. Whether those digits mean anything useful is a function of the "quality" of the meter and quality translates into $$ which, ultimately shows up as the amount of "noise" present on the signal input to the ADC chip. (That is a topic beyond our control)

Pending some research on the ADC used in the 289, working with the apparent specification seems logical: it is a 50,000 count "premium" quality meter and that implies 5.5 digits of reliable information displayed. Underlying that, one could assume that the ADC is good for six or maybe eight digits that lie beyond our reach.

It also appears that the 289 is very good at microAmp measurements. Why's that? The source of my data is a very low noise device and the plot of the measured data does not appear to be affected by noise in the meter. That it is "Fluke" implies that the measurement is reliable.

For purposes of your software, making all reliable digits available on download is essential. That means that 100 nanoAmp (7th digit after decimal) should be available. In effect, the 289 is a 7.5 digit meter in the microAmp range. That is in the specification from the manual. Also for the lowest 500 microAmp range, the resolution is good to the eighth digit at 10 nanoAmps.

If that were not enough, the 10A range is valid to the milliAmp! That is where the Fluke 289 shows its worth a premium price.

[dbbotkin]

One more thing . . . In my download script I convert MINIMUM AND MAXIMUM TO floats, sum them and divide by 2. That is my "output" so that is an extra digit some of the time. BTW, do you know anything about the "AVERAGE" data? It looks strange to me until the last sample.

[N0ury]

@dbbotkin For each measurement data returned by the DMM there is: PRIMARY, MAXIMUM, MINIMUM, AVERAGE values. They are all returned by the meter. None is calculated by the utility.

In fact, for each measurement there is a certain number of samples (the utility displays this number). For this set of data we have PRIMARY (start of event), MAXIMUM and MINIMUM. AVERAGE is probably calculated by the DMM. I think it's the mean of all samples of the period. FlukeView Forms documentation says: During interactive logging, FlukeView Forms compresses the input being measured by the meter into a series of events. Each event is represented by a sample at the start of the event, a duration, starting time, ending time, and the highest, lowest and average values detected during the event.

These four data are stored in 8 bytes (64 bits in IEEE-754 IEEE Standard for floating-point arithmetic).

[dbbotkin]

Very useful info. I agree fully, for purposes of your software, the output should be "as returned by the meter" and leave any math operations or analysis for the user to implement.

All my work in Python has been to capture data from multiple, often older, meters using the GPIB interface and VISA library for communication. Mostly for testing, verification and calibration. Normally I used Python 2.7 with IDLE on a PC in Win7, however I am currently on my MacBook and although I have 3.10 installed, I'm using Thonny as an IDE and it has 3.7 embedded. So my status is still pretty much 'debug' around getting usable data from the 289 for some experiments with "lithium capacitors" as voltage reference transfer standards due to their low leakage current.

I have not actually run your software as it is meant to operate 1. because busy and 2. I don't do command line if I can avoid it. NOW it is time to try it out!

[N0ury]

I have changed the code. I have also changed the README to say that data are returned by the DMM and not calculated or processed. commit 1c33dac v0.2.10

[dbbotkin]

"Measurement" is the average of Min and Max

[N0ury]

Very interesting, I will have learned something. I don't use this average value, I never really thought about it. This information may be useful for future questions.

[N0ury]

@dbbotkin can we close this issue now?