Treatment is false-detected as carb correction

[stefan-hausmeister]

If the carb-record is recorded in advance of the insulin_bolus the detection of a meal bolus does not work as expected. The insulin-record is being pushed to the next run and is lost as the belonging carb-record will never be found. This leads to missing insulin boli at nightscout and false-identified carb-corrections.

The service needs to look before and after the insulin-bolus? The behaviour of diasend may have changed (happened with CamAPS App and ypsopump). Unfortunately the carb record is not always in advance of the bolus, it changes without any idea why.

This is a sample dataset which belongs together:

{
    "type": "carb",
    "created_at": "2022-10-31T11:26:56",
    "value": "36",
    "unit": "g",
    "flags": []
},
{
    "type": "insulin_bolus",
    "created_at": "2022-10-31T11:26:57",
    "unit": "U",
    "total_value": 3.7,
    "spike_value": 3.7,
    "suggested": 6,
    "suggestion_overridden": "yes",
    "suggestion_based_on_bg": "no",
    "suggestion_based_on_carb": "yes",
    "programmed_meal": 6.001,
    "flags": [
        {
            "flag": 1035,
            "description": "Bolus type ezcarb"
        }
    ]
}

[burnedikt]

Thanks for opening this issue and the root cause analysis. 🙏

I've noticed this behavior as well but didn't yet get around to look into it.

Generally, either diasend is slow at processing events or camaps is sending them delayed but just like you said, sometimes the insulin bolus appears later in the api than the corresponding carbs. And by later I don't mean that it's created_date is later than that of the carbs (as is the case in your example) but that it is not returned via the API yet at e.g. 09:00, but the carbs already are and five minutes later, both are there. So even if we look before and after the carbs, this may not help. We'd need to basically delay the processing of bonus and carb events from diasend api to ensure all values are there correctly or data will be missed. This in turn goes at the expense of having real-time data as it will mean that all boli and carbs appear a few minutes later on noghtscout. CAM aps only uploads data every 5 mins to diasend which would mean we'd probably get a total delay of 10 minutes unless there are better ideas on how to mitigate this issue.

[burnedikt]

As for the specific case you described (bolus created after carbs), we could do it similarly to the bolus: if no corresponding carbs are found for a bolus, we remember the event and int he next loop we check again whether carbs are there now. However, the difference with seeing only the carbs is that in comparison to the bolus, you don't know whether it's bist a carb correction event or carbs belonging to a bolus as this kind of metadata isn't there on the carb events (for boli, it is known if there should be carbs involved or whether it's just a correction bolus). Again, this goes at the expense of a timely import of the values as we'd need to delay the processing of each carbs event without proceeding bolus.

[stefan-hausmeister]

Thank you for the quick reply and the thoughts on it. Due to the lack of having the necessary meta data, there seems to be no chance to solve this issue easily. To have a delay would not be an issue (for me), as long as the blood sugar values are not affected. On the other hand, not having the correct insulin amount is a bigger issue (for me). How important is it to identify a meal bolus? It could also be an option to store the bolus and the carbs separately if - after the next loop - the records can still not be assigned. This would be better than losing this record. Maybe it's possible to tag the record as "postponed" to handle it in the second loop as "store anyway" ;-)

[burnedikt]

This plot really thickened over the last days / weeks and it seems like diasend switched to now always having the carb record before the bolus record. Which of course doesn't work with the current logic which always expects it to be the other way around. At least this is my experience.

I can also imagine this being related to #14 which I've also seen happening repeatedly now, not only for carb corrections, but also for correction boli, slowly absorbed meals and snacks. (So basically any treatments that not a meal bolus).

[burnedikt]

Thank you for the quick reply and the thoughts on it. Due to the lack of having the necessary meta data, there seems to be no chance to solve this issue easily. To have a delay would not be an issue (for me), as long as the blood sugar values are not affected. On the other hand, not having the correct insulin amount is a bigger issue (for me). How important is it to identify a meal bolus? It could also be an option to store the bolus and the carbs separately if - after the next loop - the records can still not be assigned. This would be better than losing this record. Maybe it's possible to tag the record as "postponed" to handle it in the second loop as "store anyway" ;-)

yeah, I think that's the best and probably only way forward right now. Anyhow, I can confirm that either diasend or CamAPS changed something so that now bolus records come AFTER the carb records within the api response instead of before where we got the bolus first, carbs afterwards.

Will try to amend the logic so that if a carb event is retrieved and there's no following / preceeding bolus, it'll be "postponed" which means either keeping it in a backlog to be checked again on the next run, or proactively sending it to nightscout but then deleting it in the next run of the polling loop if it turned out to actually belong to a bolus which was not yet known in the previous run.

[stefan-hausmeister]

yeah, I think that's the best and probably only way forward right now. Anyhow, I can confirm that either diasend or CamAPS changed something so that now bolus records come AFTER the carb records within the api response instead of before where we got the bolus first, carbs afterwards.

Will try to amend the logic so that if a carb event is retrieved and there's no following / preceeding bolus, it'll be "postponed" which means either keeping it in a backlog to be checked again on the next run, or proactively sending it to nightscout but then deleting it in the next run of the polling loop if it turned out to actually belong to a bolus which was not yet known in the previous run.

Thank you so much for your awareness and having initiated this project. Everything could be so easy with some more details of the records...

[burnedikt]

Version 0.7.6 first of all addresses the issue that bolus records can now come after the carb records instead of vice versa. Both scenarios should now be supported.

However, this only holds if the batch being processed contains both the carb and corresponding bolus record (no matter which order). If, however, the processed set of records does only contain the carb record, the processing of it is not yet postponed - which means it can still happen that some treatments are falsely stored as carb corrections (since the bolus belonging to the carbs only comes in the next batch of records). Working on fixing this.

[burnedikt]

which means it can still happen that some treatments are falsely stored as carb corrections (since the bolus belonging to the carbs only comes in the next batch of records). Working on fixing this.

should now be addressed in version 0.7.8. Happy to get some feedback

[burnedikt]

Ran this for a few days now for testing purposes and couldn't reproduce the multi-import issue anymore. Closing this issue. Feel free to reopen if you still encounter issues.