Question about multiple cgm sources

[jasoncalabrese]

I'm working on adding support to Nightscout for multiple cgm sources. To do that I'm bucketing the cgm entries, and using the mean of the buckets for deltas. We're doing a similar thing in oref0.

Is Urchin doing anything like that? @scottleibrand said it seems to be working, but I was wondering how it's expected to work in this case.

Also related I've noticed that Urchin doesn't like backfilled data, I think a sort is missing.

Any links to the related code or ideas of issues I'll run in to would be great.

[jasoncalabrese]

As I expected there are some issue with multiple cgm source. when testing with https://github.com/nightscout/cgm-remote-monitor/pull/1919.

There's something assuming the number of points needed and either filtering the data, or requesting less than needed. I'm going to dig around and see if I can find it.

[scottleibrand]

Yes, @danamlewis sees the same when running two, although for us it's only missing about the first third of the CGM trendline.

[jasoncalabrese]

Depending on how long you run 2 or more the gap will grow longer

[jasoncalabrese]

Looks like part of the issue is here: https://github.com/mddub/urchin-cgm/blob/master/src/js/points.js#L15, which is then used for maxSGVCount here https://github.com/mddub/urchin-cgm/blob/master/src/js/data.js#L10

As a hack I think I can return 2x, 3x, but finding a way to do it time based would be the real fix. I'm going to experiment the hack first.

@mddub any ideas on how best to do this?

[jasoncalabrese]

This is what I'm using now, and haven't see any issues with it, but there's probably some overhead, so hopefully we can find a better way

https://github.com/mddub/urchin-cgm/compare/master...jasoncalabrese:wip/multiple-cgms

[mddub]

Urchin assumes one reading every 5 minutes. It takes the timestamp of the last reading, then makes an array of timestamps like [last_timestamp - (5 minutes) * n for n in size_of_graph], then finds the closest CGM reading to each timestamp. So if you have multiple CGM sources, the most recent reading determines the offset and thus which series is graphed. So you could see a slightly different graph on the watch depending on when Urchin happens to request data.

Urchin also uses the width of the graph to decide how much to cache, so in theory, with n CGM sources you'll only see 1/n worth of graph. In practice, it uses the timestamp of the last reading for the entries query (to minimize data transfer), so you could miss readings. Then the cache would include slightly more than 1/n (and yeah, backfilled data won't show up).

What's the motivation for multiple CGM sources? Wearing multiple sensors, or just uploading the same data from different devices? (with/without raw?)

This all feels very niche to me, but a simple solution would be to add a section to settings called "Super Advanced" which lets you specify a device name to limit the query to (share2, openaps://cgm, etc.). Does that work, or is important to have them all?

If you want them all, and want to average them instead of pick the closest one, I think it gets pretty complex. There needs to be an option to "average multiple CGM sources", and when that's enabled, either (a) disable caching and fetch full history each time, or (b) expand the cache to something like 5x and make the entries query smarter - something like

{$or: [
  {$and: [from device 1, more recent than device 1 last entry]},
  {$and: [from device 2, more recent than device 2 last entry]},
  {$and: [not from any known device, more recent than left edge of graph]}
}

converted to qs style. This assumes devices may backfill relative to other devices, but never backfill data older than what that device has already uploaded.

Here's where the bucketing and averaging would happen: https://github.com/mddub/urchin-cgm/blob/59b2291/src/js/format.js#L28-L39

Help me understand the requirements a little better, then we can figure out what should be exposed in settings.

[jasoncalabrese]

My primary motivation for this is extra redundancy, so I can have multiple uploaders (openaps rigs in my case) in different parts of the house, support for a failed rig at night, hot swap rigs, etc. It does feel niche, but I do know of several people that use a mix of share and xdrip, and now have to be careful to prevent overlaps by stoping 1 upload source before starting another.

For NS I'm trying to use the bucketing only for deltas, so all the actual values are plotted and displayed.

With my hacks https://github.com/mddub/urchin-cgm/compare/master...jasoncalabrese:wip/multiple-cgms it's mostly working. Deltas can be a little off, but do the way caching and backfilling works it's not bad.

I think any options for this are going to be confusing, so hoping we can find a way to adjust the cache/request size dynamically. Maybe the cache could truncate based on time instead of length, and the initial load could be date filtered instead of count limited.

[mddub]

@jasoncalabrese, I think #45 should make the cache hold the right number of SGVs. Can you give it a try?

What's your experience been with weird deltas? We could make the delta calculation use only SGVs from the same device as the most recent SGV, to guarantee they won't be weird. (Possibly at the cost of sometimes seeing no delta if there are gaps in one device's data.)

[mddub]

Caching by date (#45) is in the latest release. I'm not going to tackle bucketing for deltas, but I'll leave this issue open in case you want to work on that.

Here's where it would need to change: https://github.com/mddub/urchin-cgm/blob/21d71e5/src/js/app.js#L65 https://github.com/mddub/urchin-cgm/blob/59b2291/src/js/format.js#L154-L160

[jasoncalabrese]

Thanks, I think I will take at bucketing for the delta. The other option is using the new NS delta property, but I'm not sure how to start using those in a backwards compatable way.