search

NightscoutFoundation / xDrip

Nightscout version of xDrip+
https://jamorham.github.io/#xdrip-plus
GNU General Public License v3.0
1.4k stars 1.14k forks source link

Backfilled data incompletely fetched from Nightscout #1770

Closed tolot27 closed 7 months ago

tolot27 commented 3 years ago

XDrip does not fetch all backfilled data from Nightscout:

In the xDrip screenshot you can see a second gap between 10:07 and 10:52 and a third gap between 13:07 and 13:37.

The log shows the following entries:

BgReading
15:43 Cannot inject display glucose value as time difference too great: 2021-06-29 14:43:07 vs 2021-06-29 13:42:47
11:37 Cannot inject display glucose value as time difference too great: 2021-06-29 10:42:49 vs 2021-06-29 08:27:49
11:32 Cannot inject display glucose value as time difference too great: 2021-06-29 10:42:49 vs 2021-06-29 08:32:49
11:27 Cannot inject display glucose value as time difference too great: 2021-06-29 10:42:49 vs 2021-06-29 08:37:49
11:22 Cannot inject display glucose value as time difference too great: 2021-06-29 10:42:49 vs 2021-06-29 08:42:49
11:17 Cannot inject display glucose value as time difference too great: 2021-06-29 10:42:49 vs 2021-06-29 08:47:49

None of the timestamps correlate with the gaps except 2021-06-29 13:42:47 wich is the end of the second gap (first BG value).

Your environment

This issue is different to #795 because all data was sent to Nightscout.

Navid200 commented 3 years ago

Please correct if anything is wrong in the following sequence:

1- When a master disconnects from the transmitter, a gap is formed in the readings. The gap will also exist on Nightscout at the time.

2- The follower reads from Nightscout and so, ends up with the same gap.

3- The master reestablished connectivity with the transmitter. Since the transmitter sends data covering past 3 hours all the time, if the gap is within the past three hours, the gap is filled on the master. The master then sends the backfilled readings to Nightscout. So, the gap disappears on Nightscout as well.

4- The only way for the follower to fill the gap now is if it could download past data from Nightscout. Unfortunately, xDrip/Nightscout does not offer this capability (https://github.com/NightscoutFoundation/xDrip/issues/403).

That's why after buying a new phone, one cannot just download the past readings from Nightscout. We can perhaps open a feature request for adding the capability to download past readings from Nightscout.

Navid200 commented 3 years ago

I wasn't able to recreate the problem. The follower was able to get the missed readings and fill the gap. So, my description is not correct. At least, it's not complete.

Is it possible that the gap can be filled if it is not too far back in time?

tolot27 commented 3 years ago

There is a graph option to show backfilled values. Unfortunately, it does not work with in my setup. It is activated all the time but never shows backfilled data differently at the collector/share. I tried creating a screenshot to show that some backfilled data are fetched from Nightscout.

Navid200 commented 3 years ago

Sorry if you have already looked at this and are aware: https://github.com/NightscoutFoundation/xDrip/blob/14c806e9d69b71c4b6fcb62efdc673e378c0bb13/app/src/main/java/com/eveningoutpost/dexdrip/Models/BgReading.java#L243-L244

This is what triggers the logs you have. I am still trying to recreate this. I think I may be able to if I delay the recovery on the follower by 2 hours based on those logs. So, I am trying that right now.

tolot27 commented 3 years ago

Nightscout follower is indeed able to fetch the data. The following screenshot is from my dev phone which is also configured as NS follower but was offline some days. Switching it on, immediately fetched the data including backfills:

Navid200 commented 3 years ago

Something caused the ratelimit in the if statement to become true. If we find that, we find what triggers the issue.

tolot27 commented 3 years ago

I've extracted the relevant BG values from the database and highlighted the gaps. Looks like gaps are filled only the last 35 to 40 minutes. Backfillgap

Interestingly, there is no gap between 08:17:49 and 09:27:49.

tolot27 commented 3 years ago

Sorry if you have already looked at this and are aware: https://github.com/NightscoutFoundation/xDrip/blob/14c806e9d69b71c4b6fcb62efdc673e378c0bb13/app/src/main/java/com/eveningoutpost/dexdrip/Models/BgReading.java#L243-L244

I don't think this is related to this issue because the if BG values at that time are fetched from NS.

Navid200 commented 3 years ago

That is interesting. If the follower can download past readings from Nightscout, the master should be able to do it too. But, it cannot. I'd like to know why. But, that is not related to this issue.

adamecker commented 2 years ago

I've seen this issue happen a few times recently. Usually when collector goes offline, then comes back on and uploads to nightscout. Follower only gets new data from that point on even though nightscout is backfilled from the collector

tolot27 commented 2 years ago

I've seen this issue happen a few times recently. Usually when collector goes offline, then comes back on and uploads to nightscout. Follower only gets new data from that point on even though nightscout is backfilled from the collector

Can you reproduce the problem 100%? Do you discover something like a backfill window of certain minutes like I did above or is there no backfill data at all?

tzachi-dar commented 2 years ago

Here is my 2 cents regarding this issue. (all from the top of my head without checking in the code.)

1) xDrip remembers the last time that it was doing the last read. (In other words, if xDrip is reading now, and the last read is from 10:00:00, than xDrip will read again asking for data from 10:00:00 and after it).

2) In some cases writers to nightscout don't write in the "correct" order. That is, if there was a GAP xDrip might write the data of 10:00 followed by the data of 9:40, 9:45. 9:50, 9:55.

From 1,2 if there is a gap, and the writer has just written the data of 08:00 - 9:40 and 10:00, the reader can get the data with the gap. Since the reader has the last reading set to 10:00 it will not ask for data before it. Now the writer will close the gap, but this will not help the reader.

This explains why the problem only rarely happens. HTH and hope I remember it right.

adamecker commented 2 years ago

@tzachi-dar I believe you are correct. This happens when my collector is broken for reasons that's aren't worth explaining here. Then when it's fixed it uploads the most recent data to nightscout first. The follower gets that point and then the collector back fiils the data but the follower never looks before it's most recent data point

Navid200 commented 7 months ago

I'm not sure if we need to fix this.

I use Nightscout as a repository of all the readings. To access that, for example to create a report, I use a browser to access Nightscout.

Using Nightscout to follow allows one to get alerts if something is wrong. As long as the follower can do that, I don't see why the follower needs a backfill of all past readings.

Please let me know if there is a reason you need the follower to get a backfill for all past readings and we can reopen.