Show raw data in main graph

[zjedr]

I really like the raw data in NS, an option to show raw data on the xDrip main graph for all data sources that provide it would be great (at least for xBridge Wixel).

[AdrianLxM]

xBridge/Wixel uses just the unfiltered data at the moment. All dots you see on the graph when this data source is selected (apart from manual calibration entries) is "raw" :)

[zjedr]

Thanks @AdrianLxM, but that's not what I'm seeing when I compare xDrip to NS. As you can see from these screenshots, the NS shot shows raw data that is not on the xDrip graph. For example the 8:08 reading shows BG of 4.7/ raw 5.9 on NS but only shows the BG reading of 4.7 on xDrip (sensor is on its last legs). I would like to see both readings in xDrip as it is in NS.

[tzachi-dar]

Can you please explain your setup? Is xdrip the one collecting the data? does it do so using BT (with xDrip or xBridge)?

Thanks Tzachi

On Tue, Nov 3, 2015 at 2:11 PM, zjedr notifications@github.com wrote:

Thanks @AdrianLxM https://github.com/AdrianLxM, but that's not what I'm seeing when I compare xDrip to NS. As you can see from these screenshots, the NS shot shows raw data that is not on the xDrip graph. For example the 8:08 reading shows BG of 4.7/ raw 5.9 on NS but only shows the BG reading of 4.7 on xDrip (sensor is on its last legs). I would like to see both readings in xDrip as it is in NS.

[image: raw2] https://cloud.githubusercontent.com/assets/13398921/10907590/7cb9cea8-827f-11e5-8d2f-3469cc6f6009.jpg [image: raw1] https://cloud.githubusercontent.com/assets/13398921/10907589/7cb8d94e-827f-11e5-8992-e1db1c6dab31.jpg

— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/issues/172#issuecomment-153335019 .

[zjedr]

Hardware data source is "xBridge wixel". I run the xBridge2 version of the xDrip-wixel hardware. Data is only collected via xDrip

[AdrianLxM]

@zjedr, in this case, the coloured dots are "raw" on both xDrip and NS. The Nightscout site does do a "trend prediction" with the raw values, if the data is generated by the raw values. This can happen with the official receiver as well, if no noise is detected.

Example of what Nightscout does:: Value "raw": 100 Value "filtered": 90 Value shown: 100 Raw on website: 100*(100/90) = 111 (also the white dots)

As xBridge always calculates the values with raw input, the white dots on Nightscout will always be a prediction, not a real raw value.

[AdrianLxM]

To be more clear, the formula is: raw * (estimatedBG / filtered) = "raw shown on website"

This means, if the official receiver detects noise and chooses the filtered value to be the one shown, you will get 100*(90/90) as result for the example above.

[jstevensog]

Also, at the start of the sensor, there is a factor applied to the BGL value. This is the same as Dexcom. So, for the first 24-48 hours, NS will show a fairly large difference between "raw" and "actual". This factor is meant to deal with the sensitivity change over time of a new sensor.
Raw data in NS is the value from the sensor converted to BGL using a different algorithm in NS as to what xDrip uses. The value to Dexcom receiver/uploader users, is that it shows how Dexcom uses the " filtered" value if it doubts the raw value, and will display BGL trend when the reciever gives up and shows blanks. XDrip doesn't use the filtered value and always shows a raw value it captures. Cheers

---- AdrianLxM wrote ----

To be more clear, the formula is: raw * (estimatedBG / filtered) = "raw shown on website"

This means, if the official receiver detects noise and chooses the filtered value to be the one shown, you will get 100*(90/90) as result for the example above.

— Reply to this email directly or view it on GitHub.

[AdrianLxM]

Also, at the start of the sensor, there is a factor applied to the BGL value. This is the same as Dexcom. So, for the first 24-48 hours, NS will show a fairly large difference between "raw" and "actual".

In the recent version this is solved: raw and filtered raw both adjust the same way: https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/commit/1ebd1cc552e370a57f4ed0cdce89ed96e3ef1041 https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/commit/c1524a369a2fcede5c0ac6df9a33bad246fb79d1

[jstevensog]

Hmmm. That is not my experience. I still see the "raw" value in NS being higher than the "displayed" BGL, and I didn't think we were doing anything with the "filtered" value, as wixel-xDrip and most other collectors don't get it. Maybe we are getting the terms mixed up.

Dexcom sends two values to the receiver. The first we call "raw" and the second we call "filtered". In xDrip, if I understand correctly (not familiar with the actual code) we only use the "raw" value to calculate BGL. We store "raw" and "filtered" as the same value and send the same value to NS. Might be different with xBridge and Share.

But NS displays the calculated BGL reported (SGV?), and if you turn it on, it also displays the "raw" value BGL. The "raw" value BGL is calculated in NS as the reported "raw" value using the last calibration entry data, if I understand it correctly.

In the case of xDrip, and Dexcom, the "displayed" BGL is calculated from the calibrations to give slope and intercept. Also, in the first 24-48 hours it calculates it with a factor that allows for rapidly decreasing sensor sensitivity after insertion. So at the beginning of a sensor NS displays significantly different "raw" and "displayed" BGL values, which decrease over time.

In the case of the Dexcom, there is also the added issue of it selecting either the "raw" or "filtered" value from the transmitter to use to calculate the "displayed" BGL, and also the fact that it will display nothing if it thinks the sensor data is bad.

NS then displays the "raw" BGL based on the last calibration to "fill the gap".

So, regardless of how we deal with "raw" and "filtered" values in xDrip, NS calculates and displays "raw" BGL based on last calibration data as I understand it.

The value of displaying "raw" BGL in NS when using xDrip is very suspect. I do it for interest sake only, because it is a feature of NS. But I find no real value as most of the time the "raw" and "displayed" BGLs are either tracking the same, or identical, depending on the last calibration.

With xDrip, the "displayed" BGL is always more accurate than the "raw" BGL, except perhaps at sensor start. But at that point, neither xDrip or Dexcom can be relied upon as there is insufficient calibration data to accurately calculate "displayed" BGL. Hence why I do a third calibration an hour or two after the double calibration. I get pretty good accuracy in "displayed" BGL within the first 24 hours that way.

Hope this clears up my comments. Cheers

On Wed, Nov 4, 2015 at 8:01 AM, AdrianLxM notifications@github.com wrote:

Also, at the start of the sensor, there is a factor applied to the BGL value. This is the same as Dexcom. So, for the first 24-48 hours, NS will show a fairly large difference between "raw" and "actual".

In the recent version this is solved: raw and filtered raw both adjust the same way: 1ebd1cc https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/commit/1ebd1cc552e370a57f4ed0cdce89ed96e3ef1041 c1524a3 https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/commit/c1524a369a2fcede5c0ac6df9a33bad246fb79d1

— Reply to this email directly or view it on GitHub https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/issues/172#issuecomment-153486698 .

John Stevens "You are how you live, not what you have."

[AdrianLxM]

In xDrip, if I understand correctly (not familiar with the actual code) we only use the "raw" value to calculate BGL/SGV. We store "raw" and "filtered" as the same value and send the same value to NS. Might be different with xBridge and Share.

There is a difference in xDrip and xBridge: Even though the calculatedBGL/SGV is the same (calculated from raw), it stores and uploads the "raw" and the "filtered" value.

But NS displays the calculated BGL reported (SGV?), and if you turn it on, it also displays the "raw" value BGL. The "raw" value BGL is calculated in NS as the reported "raw" value using the last calibration entry data, if I understand it correctly.

It is not only calculated from the reported "raw" value and the last calibration entry data, but also the reported SGV is part of the formula.

 var ratio = cleaned.scale * (cleaned.filtered - cleaned.intercept) / cleaned.slope / sgv.mgdl;
 raw = cleaned.scale * (cleaned.unfiltered - cleaned.intercept) / cleaned.slope / ratio;

https://github.com/nightscout/cgm-remote-monitor/blob/master/lib/plugins/rawbg.js#L59

This "ratio" leads to the phenomenon that the "Nightscout-raw" is shown even higher (by the ratio "raw/filtered") if the SGV is calculated from the raw value and "raw">"filtered". "Nightscout-raw" is lower than "raw wit calibration" if "raw<filtered", if SGV is calculated from raw.

The value of displaying "raw" BGL in NS when using xDrip is very suspect. I do it for interest sake only, because it is a feature of NS. But I find no real value as most of the time the "raw" and "displayed" BGLs are either tracking the same, or identical, depending on the last calibration.

The fromula NS uses in this case is a prediction/heuristic into the future. It is not the real raw value as we understand it (even with added calibration). I believe this to be very confusing as well. But once you know that, you can use it to find trends faster.

In the case of xDrip, and Dexcom, the "displayed" BGL is calculated from the calibrations to give slope and intercept. Also, in the first 24-48 hours it calculates it with a factor that allows for rapidly decreasing sensor sensitivity after insertion. So at the beginning of a sensor NS displays significantly different "raw" and "displayed" BGL values, which decrease over time.

In current master (since late September/early October), this should no longer be the case, as both "filtered" and "unfiltered" values get "age adjusted" the same way. Here is the pull request including a similar discussion: https://github.com/StephenBlackWasAlreadyTaken/xDrip-Experimental/pull/149

Maybe we should discuss the topic with "over/under-estimation"=prediction of raw values with the Nightscout team?

cheers, Adrian

[zjedr]

Thanks for the explanations, believe it or not, it does make more sense now. It also partially explains my other ticket https://github.com/StephenBlackWasAlreadyTaken/xDrip/issues/115 (probably should be closed) where NS-raw and BGL was vastly different on a new sensor, I would have been on an old release then and didn't have the same issue on my last sensor.

Whatever the NS-raw figure is, I do find it useful to see when a trend is peaking as it gets closer to the BGL. It's visually easier to pick than looking for a slight change in the g-worm trajectory.