search

nightscout / cgm-remote-monitor

nightscout web monitor
GNU Affero General Public License v3.0
2.38k stars 71.59k forks source link

/entries/history does not return all the available items #8090

Closed justmara closed 12 months ago

justmara commented 12 months ago

Describe the bug I'm currently testing AAPS dev-l and encountered strange behaviour with NS API v3 client - it fails to backfill BG data from NS. As I can see from logs NS client queries /entries/history?from={}&limit={} endpoint with limit 0f 500 items and on whenever it receives will 500-items long recordset it believes that there is more data and makes new query, setting last received item's timestamp for from value. But for some reason NS not always returns all the 500 records even when there are much more available in database.

To Reproduce Clean install AAPS 3.2 dev-i, setup NSClientV3, enable BG backfill from NS and press 'Full synchronization' in NSclientv3. I've started full sync in AAPS dev-i build and sow this in log --> GET https://[my ns here]/api/v3/entries/history/1694366629182?limit=500 --> GET https://[my ns here]/api/v3/entries/history/1694366929102?limit=500

and this in NSClient log. image

So on second iteration it received only 497 items of requested 500 and thought that the data is over.

But I've modified AAPS code a bit so it would stop only if received array size is smaller than *0.95 of requested. And got this log

image

So there was much more data in NS, but many times NS refused to return full 500-records long response.

Looks like if limit were applied on db read, but then some records were removed from response.

Expected behavior

Request with limit=500 option should always return 500 records, if there are more 500 or more available in database for that query.

Your setup information

sulkaharo commented 12 months ago

The entries also contains glucometer readings (type = mbg) so if you're storing these and want to download just cgm records, you should ask for just type=sgv records. Or... I'm assuming the reason for getting less than 500 is that you're actually getting 500 records at the time and some are not sgvs.

justmara commented 12 months ago

Hmmm, sounds reasonable. Thats why I've asked this in AAPS channel first :) But there I was said that this issue must be created here :) And now I see, where those records are filtered-out in nsclientv3. Thank you. Going back to AAPS channel...

sulkaharo commented 12 months ago

Btw does NSClientV3 use the rest api or the websockets? We'll probably remove a lot of the websocket support in the next release so at least old NSClient versions will stop being working.

justmara commented 12 months ago

It seems to be able to use both, but I've only got rest mode to work at acceptable level.