Excessive profile syncing to Nightscout

[sulkaharo]

Hi! I got reports from some Loop/FreeAPS users that they're having issues with Nightscout profiles. Checking their profile data, I noticed they have a huge amount of profile data in their system, where in some cases the profile has been synched in three times within one second, resulting in the profile collection bloating to be huge (in one case, recent profiles resulting in a 33 MB JSON object).

I doubt the profile actually changes this often, so I wonder where the glitch is that causes this. One suspicious part of thee data was that the loopSettings section seems to have "scheduleOverride" section, which seems to be part of the runtime data and not part of the profile, where if the next scheduled basal change is sometimes entered into the profile data and sync is caused by a difference in the profile, this would trigger an unintended profile sync. I'll check if there's a simple fix for the database code in NS, but checking if there's an issue in Loop would be great.

[bartsopers]

I actually also experienced this on AndroidAPS yesterday night. If it helps, I think I can provide you with logs.

[loyalchallenger]

mLab mondgoDB is being discontinued on Heroku. What else can I use? I’ve been unable to run autotune reports because I think nightscout is not storing my data. Thanks!

[sulkaharo]

@loyalchallenger see https://github.com/nightscout/cgm-remote-monitor/wiki/mLab-discontinuation-FAQ

[loyalchallenger]

@loyalchallenger see https://github.com/nightscout/cgm-remote-monitor/wiki/mLab-discontinuation-FAQ

Thank you so much!

[loyalchallenger]

@sulkaharo I think I broke it. I updated Nightscout like it said and did everything. But now Nightscout is saying my API secret is incorrect even though it is correct. I tried all the tips I found on google, but nothing is working.

[loyalchallenger]

I was able to get nightscout back up and migrate to Atlas. I still can’t get autotune to run though.

[Nightfoxy]

Jon Fawcett has created change that reduces the profile updates, but isn't a complete fix. The basic issue is that loop uploads a new profile whenever any settings are changed or overrides are run or modified. This can add up quickly, even if the settings in NS are not what was updated.

Its causing issues on my NS site as well. I don't know of a quick way to purge old profiles.

https://github.com/LoopKit/Loop/pull/1283/commits/06b8a116eb475401c138042fee84ef600800f658

[sulkaharo]

You mean a new profile is sent every time Loop sets a basal rate? The profile data is very much not intended for that purpose - it should only be used to store the long term dosing profile. There's multiple other ways to store temporal data, please describe the scenarios to me that you'd like to store and I can help figure out something that works in concert with Nightscout design.

[Kdisimone]

@sulkaharo the problem is that loop doesn't save profiles itself for users to switch between. So it's a little hampered of a system to integrate with NS yet in a way that would be most clean. Ideally, the integration would be like remote overrides are done currently. Loop saves profiles, those are uploaded to NS and the looper or a NS user (with api authorization) could switch between profiles like you can on most pumps.

Without loop saving profiles, the benefit of loop uploading current schedule for basal, cr, and isf is that a remote parent can have a look at existing settings before making suggestions to school nurse or Looper about a direction to change settings as needed. Unfortunately, that benefit is limited to CURRENT settings, so loop does need to upload every time those would change in loop settings by the user

[sulkaharo]

If I understand correctly what the data is, the correct way to store that would be a devicestatus record. That collection is intended for the temporal data and the Loop pill will have access to the recent statuses at all times in a way that scales. This is what Openaps and Androidaps uses, in addition to treatment records. I understand the desire to communicate the information but as is, the way Loop uses the profiles breaks Nightscout and with the current architecture and storage limits most users have, there's no simple way to chance Nightscout to support the way Loop works right now.

[sascha-egerer]

@Nightfoxy @loyalchallenger

I've just deleted all the profiles to get Autotune working. You can also do that by running this script on your nightscout page. BUT BE AWARE! THIS WILL DELETE ALL YOU PROFILES BUT THE LAST ONE!

1. Use Chrome and open your nightscout
2. Open the Console to run JavaScript (https://developers.google.com/web/tools/chrome-devtools/open#console)
3. Paste this script into the console input field at the bottom of you browser

   var url = window.location.protocol + '//' + window.location.host;
   fetch(url + '/api/v1/profile.json', {
   credentials: 'same-origin',
   headers: {'api-secret': localStorage.apisecrethash}
   }).then(response => response.json()).then(data => {
   data.forEach(function (item, index) {
       if (index > 0) {
           fetch(url + '/api/v1/profile/' + item._id, {
               method: 'DELETE',
               credentials: 'same-origin',
               headers: {'api-secret': localStorage.apisecrethash}
           }).then(response => response.json());
       }
   });
   });

4. BE AWARE IT WILL DELETE ALL YOU PROFILES!
5. Hit Enter

[loyalchallenger]

@sascha-egerer , I had the time zone wrong in Atlas. 🤦‍♀️Mine is working now. Thank you!

[sulkaharo]

Related ticket in FreeAPS https://github.com/ivalkou/LoopWorkspace/issues/6

[sulkaharo]

Would love some suggestions from Loop developers on how to fix this. @ps2, can you point someone at this? I'm in a slightly difficult position here, where I don't want to break NS for Loop users, but simultaneously the current Loop behaviour is breaking NS for said users who automatically assume this is a bug in Nightscout which adds to NS support load, plus this is possibly causing issues (maybe even wrong dosing suggestions) for Loop users who use Autotune, so something needs doing. Given I don't use Loop and don't have any other developers actively participating in Nightscout development, it's even hard to test any potential performance improvements that could mitigate this.

[ps2]

There are two different things that are causing updates to the profile 1) Any Loop Settings change, including delivery limits, override/preset definitions, deviceToken (for push notifications), etc. 2) When a user enables/disables an override/preset.

​2: should definitely change to some place other than the profile. When I updated the initial overrides PRs to actually get them into Loop, I started this process; there is also information about the currently running override in devicestatus.json, and IIRC, I have the NS code looking there when it needs to display overrides. So we could probably drop the profile update that is triggered when someone enables or disables an override. The point Katie mentions above about a caregiver knowing what the currently running schedule is, might be tougher to resolve. But regardless, I do agree we should remove this from the profile.

​1: (Loop Settings). If we don't put these in the profile, then where should they go? Loop settings are expanding and including more details in the code that will be coming from Tidepool. It would be great if NS could support storing these fully. It would open the door to someone restoring Loop settings to a new phone from NS.

[sulkaharo]

Hi! Thanks for that explanation, this makes a lot more sense now. Also sounds like the case where multiple profiles are synched in very rapid succession (up to multiple times / second) sounds like a bug, where the internal state of the profile is being synched while it's being changed? I'd assume people don't change the profiles very often, so the #1 scenario shouldn't be problematic in terms of data storage, but given the code in the profile system doesn't except to see any of the Loop specific data, it'm assuming it's possible there could be unexpected results if someone for example uses the profile editor on a profile from Loop so I think on long term we need something better.

There's been multiple discussions earlier on adding an API to store settings objects from external apps. This is not a super hard thing to do - do you have any requirements beyond having an API that allows for saving and querying app specific settings and this being available in the NS client?

[sulkaharo]

Ok related - MongoDB Atlas folks are saying at least one Loop user has had to switch out from Atlas due to his/her data usage exceeding 120 GB / week and thus blowing the usage cap of the free Atlas tier. Apparently the most problematic queries issued by Loop are related to device status syncs, so would be great if someone had the time to send me samples of all the queries Loop is sending the the REST API so we can check how this could be improved.

[loyalchallenger]

Mine appears to be broken also. I’m happy to send anything, but I don’t know how...

[sulkaharo]

Some more data from Atlas: the device status syncs from Loop appear to be both too frequent and contain too much data. Namely the BG predictions in the status includes thousands of values, most of which are probably so far in the future that the data is not really used for visualisation or alarms. Apparently the last couple days' of Loop status weighs about 4.5 MB, which given Nightscout architecture of Mongo polling amounts of about 30 GB of data transfer on a monthly basis. I'll see if I can reduce the amount of data being loaded in NS side, but it'd be extremely helpful if someone could spend some time optimising the Loop REST API use in terms of data size and update frequency for all data types.

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.