Basal Profile JSON appears to be incompatible with Nightscout on Safari and iOS

[porkupan]

[From Facebook comment] Safari and iPhone's JS interpreter cannot digest the pump basal profile saved from the App via REST API. The App saves the Basal Profile in the format:

"mills": "1485461377832",
"startDate": "2017-01-26T15:09:37-0500",
"units": "mgdl"

When I save the profile from the NS site's Profile Editor, the format ends up like this, and the site is happy:

"mills": "NaN",
"startDate": "2017-01-26T20:09:00.000Z",
"units": "mg/dl"

By manually changing the startDate string in the Mongo DB from one format to the other I was breaking and fixing the working of Basal Profiles. The "mills" and "units" didn't appear to make any difference.

By further digging I was able to see that the timezone offset was the issue. I've tried many variants with the timezone offset, and neither one worked. So you probably want to specify GMT/UTC time there. For instance, this variant does work:

"startDate": "2017-01-26T20:09:00"

BTW, what's the deal with 2017 there? Is it the pump that's feeding us the wrong year?

Your Environment

• Uploader Version Number: 600
• Your Android Phone Model Name (e.g. Samsung J1 Ace): Moto G, Samsung S4
• Android Version Number (e.g. use 4.4 for 4.4.4): doesn't matter
• Network Connection at the time the issue occurred (Wi-Fi, Mobile): Wi-Fi
• (if relevant) Nightscout website version (e.g. 0.8.4 Funnel cake, master / dev branch): The latest dev version 0.10.3-dev-20171205

Brief Explanation of Issue

Basal profile reverts to default (Basal 1) on iOS due to unexpected startDate format

Uploader Error Code

None #### Steps to Reproduce (for bugs)

1. Just retrieve/save the basal profile from the app
2. Observe the Nightscout on iOS or Mac Safari revert to default basal profile
3. You can go to profile editor, observe the "unrecognized start date"

Severity Score

1.5

[Bal00]

I have also observed issues with basal rendering on Firefox on Windows 10.

On 29 Jan 2018, at 02:41, porkupan notifications@github.com wrote:

[From Facebook comment] Safari and iPhone's JS interpreter cannot digest the pump basal profile saved from the App via REST API. The App saves the Basal Profile in the format:

"mills": "1485461377832", "startDate": "2017-01-26T15:09:37-0500", "units": "mgdl" When I save the profile from the NS site's Profile Editor, the format ends up like this, and the site is happy:

"mills": "NaN", "startDate": "2017-01-26T20:09:00.000Z", "units": "mg/dl" By manually changing the startDate string in the Mongo DB from one format to the other I was breaking and fixing the working of Basal Profiles. The "mills" and "units" didn't appear to make any difference.

By further digging I was able to see that the timezone offset was the issue. I've tried many variants with the timezone offset, and neither one worked. So you probably want to specify GMT/UTC time there. For instance, this variant does work: "startDate": "2017-01-26T20:09:00" BTW, what's the deal with 2017 there? Is it the pump that's feeding us the wrong year?

Your Environment

Uploader Version Number: 600 Your Android Phone Model Name (e.g. Samsung J1 Ace): Moto G, Samsung S4 Android Version Number (e.g. use 4.4 for 4.4.4): Network Connection at the time the issue occurred (Wi-Fi, Mobile): Wi-Fi (if relevant) Nightscout website version (e.g. 0.8.4 Funnel cake, master / dev branch): The latest dev version 0.10.3-dev-20171205 Brief Explanation of Issue

Uploader Error Code

Steps to Reproduce (for bugs)

Just retrieve/save the basal profile from the app Observe the Nightscout on iOS or Mac Safari revert to default basal profile You can go to profile editor, observe the "unrecognized start date" Severity Score

1.5

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[porkupan]

startDate in the format: "startDate": "2017-01-26T20:09:00-05:00" appears to also work. Which seems to be the standard for specifying the TZ per JavaScript documentation.

[Pogman]

Please verify that your reports are being generated correctly with that change to the startdate.

There are issues in nightscout with profile handling and it will only process profile data for reports from the first group and if that startdate is not offset it will ignore any profile change treatments. To get around this there are options in the uploader to "use a single group" and "offset start date" (the defaults).

[porkupan]

If I understood the NS cgm-remote-monitor code correctly, it simply uses the standard Date constructor to create a Date object out of startDate string:

function init (profileData) {

  var profile = { };

  profile.timeValueCache = new NodeCache({ stdTTL: 600, checkperiod: 600 });

  profile.loadData = function loadData (profileData) {
    if (profileData && profileData.length) {
      profile.data =  profile.convertToProfileStore(profileData);
      _.each(profile.data, function eachProfileRecord (record) {
        _.each(record.store, profile.preprocessProfileOnLoad);
        record.mills = new Date(record.startDate).getTime();
      });
    }
};

So the most common standard for the Date string formatting should be used to be universally acceptable.

[porkupan]

Yes, the reports seems to be generated correctly. The start date for basal profile probably doesn't matter as long as it's somewhere in the past. No idea why the year comes out as 2017 however. Couldn't find any code that would force it somehow (and I verified that both the pump and the phones are on 2018).

[Pogman]

The option offsets by 12 months to cover a reasonable period.

[porkupan]

Oh thanks, I get it now!

I am still thinking it would be nice to try and get all the "dates" and timestamps in the app to conform with the same standard. I see that some timestamps are in the form of a unix date string, while the others are in the [almost] ISO standard.

{
    "_id": {
        "$oid": "5a6d67fa92fd2200135fd9b9"
    },
    "created_at": "2018-01-28T01:04:42-0500",
    "device": "medtronic-600://6211-1519065",
    "pump": {
        "battery": {
            "percent": 75
        },
        "clock": "2018-01-28T01:04:42-0500",
        "iob": {
            "bolusiob": 0,
            "timestamp": "Jan 28, 2018 1:04:42 AM"
        },
        "reservoir": 113,
        "status": {
            "bolusing": false,
            "status": "normal :::. 7h10m",
            "suspended": false
        }
    },
    "uploaderBattery": 50
}

This app's CGM entry:

{
    "_id": {
        "$oid": "5a6390b61280607adeb8296a"
    },
    "type": "mbg",
    "dateString": "Sat Jan 20 13:55:28 EST 2018",
    "device": "medtronic-600://6211-1519065",
    "mbg": 194,
    "date": 1516474528561
}

The XDrip:

{
    "_id": {
        "$oid": "5a3ba551820dc160ee8bda6d"
    },
    "device": "xDrip-DexcomShare",
    "type": "cal",
    "date": 1513857950314,
    "dateString": "2017-12-21T07:05:50.314-0500",
    "slope": 661.9520286040622,
    "intercept": 43224.9300292993,
    "scale": 1,
    "sysTime": "2017-12-21T07:05:50.314-0500"
}

I've been trying to manually clean the old entries from the database, and it makes it a bit difficult to soft by dateString. :)

[porkupan]

I haven't checked the code, but I suspect the cgm-remote-monitor uses the UTC-milliseconds-since-the-epoch "date" field for identifying the events, and ignores the "dateString". Otherwise we would be seeing the same issue in the [non-]conforming browsers.

[Pogman]

This is what android supports for the api we target and what has been in use from day one. This will produce TZ format as -0800.

Everything should be handling -08 or -0800 or -08:00 without issue as that's the ISO_8601 standard... but yeah standards and reality.

For v0.6.1 I'm willing to make the dates in the profile be reworked as -08:00 but not the others without a full testing cycle (even though it should not be an issue, it's just that what's there is know to work as is).

[Pogman]

You might have noticed that manual entries via NS site will put a 'Z' on the end of various dates. I don't know but that looks like a NS bug too.

[porkupan]

Actually, this seems to be within the standard for the ISO dates. Z means "zulu" time, which is UTC.

ISO dates can be written with added hours, minutes, and seconds (YYYY-MM-DDTHH:MM:SSZ): Example var d = new Date("2015-03-25T12:00:00Z");

Date and time is separated with a capital T. UTC time is defined with a capital letter Z. If you want to modify the time relative to UTC, remove the Z and add +HH:MM or -HH:MM instead: Example var d = new Date("2015-03-25T12:00:00-06:30");

[porkupan]

I am by no means an expert in either Java or JS, so I am not sure if the TZ offset in the date string is really a standard for Android. Looking the the Java documentation it seems that you can specify pretty much any format you like. The format specifier 'Z' seems to generate -0500, but you can use the 'XXX' instead to get -05:00. Or just specify the TZ as 0 to produce the letter Z.

For formatting, if the offset value from GMT is 0, "Z" is produced. If the number of pattern letters is 1, any fraction of an hour is ignored. For example, if the pattern is "X" and the time zone is "GMT+05:30", "+05" is produced.

For parsing, "Z" is parsed as the UTC time zone designator. General time zones are not accepted.

[Pogman]

What you referenced is not supported in the android api's we target only the more recent ones.

[Pogman]

The simplest thing is to wrangle the api date data and shove a colon ":" into it.

[porkupan]

Well, the simplest is probably just not to specify the Local.getDefault() as the input into the SimpleDateFormat. I assume the function will not generate the timezone offset then. You really don't want to modify the string directly after it has been created.

I am a bit lost on why the NS seems to be able to digest the Temp Basals and other treatments, which specify the 'created_at' in the same seemingly incorrect format. I wonder if they may be using a different parser function to parse the times in the treatments entries.

{
    "_id": {
        "$oid": "5a692c161280607adec6f9ba"
    },
    "absolute": 3.3,
    "created_at": "2018-01-24T19:59:15-0500",
    "duration": 5,
    "eventType": "Temp Basal",
    "key600": "MB80099CB6",
    "notes": "microbolus 0.275U"
}

[porkupan]

I was just looking at the MongoDB entries produced by the OpenAPS, and it seems to always use the format with a ':'

{
    "_id": {
        "$oid": "5a4bef351280607ade72b48c"
    },
    "duration": 30,
    "raw_duration": {
        "_type": "TempBasalDuration",
        "_description": "TempBasalDuration 2018-01-02T15:40:06 head[2], body[0] op[0x16]",
        "timestamp": "2018-01-02T15:40:06-05:00",
        "_body": "",
        "_head": "1601",
        "duration (min)": 30,
        "_date": "06680f4212"
    },
    "timestamp": "2018-01-02T15:40:06-05:00",
    "absolute": 3.35,
    "rate": 3.35,
    "raw_rate": {
        "_type": "TempBasal",
        "temp": "absolute",
        "_description": "TempBasal 2018-01-02T15:40:06 head[2], body[1] op[0x33]",
        "timestamp": "2018-01-02T15:40:06-05:00",
        "_body": "00",
        "_head": "3386",
        "rate": 3.35,
        "_date": "06680f4212"
    },
    "eventType": "Temp Basal",
    "medtronic": "mm://openaps/mm-format-ns-treatments/Temp Basal",
    "created_at": "2018-01-02T15:40:06-05:00",
    "enteredBy": "openaps://medtronic/522"
}

[porkupan]

I have to assume that NS must be playing some tricks with the treatments' timestamps. It seems to be willing to take the Temp Basal from the App, and somehow render it properly, even though the only timestamp that appears in these entries is the 'created_at' in the format that iOS cannot handle. Looking in the cgm-remote-monitor source I don't see anything other than the Date object constructors to handle these time/date strings. Which leads me to believe that perhaps the actual basal curve rendering is done on the server side, and the client browser just displays the 'picture' of the basal curve. However, the client is doing the basal profile selecting, and that fails due to dateString format parsing. The NodeJS runtime must be able to parse all formats of dateString while the client browser is more picky.

[Pogman]

It's pretty simple ;)

Anyhow I'll put this mod across the board for all the NS data not just profiles and we'll RC v0.6.1 for a few days before release.

[porkupan]

Sounds great! Then maybe I will propose a couple of more enhancements and you consider them for implementation in the 0.6.2?

1. In the uploaderBattery updates specify the actual name of the device:

   {
   "_id": {
       "$oid": "5a46569d9b7fa6001321196c"
   },
   "device": "LGE Nexus 5X",
   "uploader": {
       "battery": 90
   },
   "created_at": "2017-12-29T14:52:13.811Z"
   }

   Would be useful to identify which actual device is currently being used to upload data (if we enable the tooltip in the "Uploader battery" pill).

2. Much more interesting, try to add the OpenAPS-like algorithm to calculate the predicted/eventual BGs array. This may enable the OpenAPS plugin to generate the BG forecast curve, which is a pretty useful tool for people who want to see where the BG is actually going given current IOB and other parameters. Here is an example of the predBGs array generated by the OpenAPS:

   {
   "_id": {
       "$oid": "5a46231a9b7fa6001321190b"
   },
   "device": "openaps://OpenAPS",
   "openaps": {
       "iob": {
           "iob": -0.04,
           "activity": -0.0043,
           "bolussnooze": 0,
           "basaliob": -0.04,
           "netbasalinsulin": -0.75,
           "hightempinsulin": 0.3,
           "timestamp": "2017-12-29T11:08:00.000Z"
       },
       "suggested": {
           "temp": "absolute",
           "bg": 114,
           "tick": "+1",
           "eventualBG": 113,
           "snoozeBG": 113,
           "predBGs": {
               "IOB": [
                   114,
                   115,
                   115,
                   115,
                   116,
                   116,
                   116,
                   117,
                   117,
                   117,
                   117,
                   117,
                   117,
                   117,
                   117,
                   117,
                   116,
                   116,
                   116,
                   116,
                   116,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   115,
                   114
               ]
           },
           "COB": 0,
           "IOB": -0.04,
           "reason": "COB: 0, Dev: -3, BGI: 0.94, ISF: 44, Target: 109; Eventual BG 113 > 99 but Min. Delta 0.50 < Exp. Delta 0.7, temp 1.1 ~ req 1.1U/hr",
           "timestamp": "2017-12-29T11:09:32.000Z"
       },
       "enacted": {
           "bg": 114,
           "temp": "absolute",
           "snoozeBG": 118,
           "recieved": true,
           "rate": 1.1,
           "reason": "COB: 0, Dev: -2, BGI: 1.22, ISF: 44, Target: 109; Eventual BG 118 > 99 but Min. Delta 0.00 < Exp. Delta 0.7; setting current basal of 1.1 as temp",
           "COB": 0,
           "eventualBG": 118,
           "timestamp": "2017-12-29T10:57:17.000Z",
           "duration": 30,
           "tick": "+0",
           "IOB": -0.146
       }
   },
   "pump": {
       "clock": "2017-12-29T06:08:00-05:00",
       "battery": {
           "status": "normal",
           "voltage": 1.3
       },
       "reservoir": 151.4,
       "status": {
           "status": "normal",
           "bolusing": false,
           "suspended": false,
           "timestamp": "2017-12-29T11:09:27.000Z"
       }
   },
   "mmtune": {
       "scanDetails": [
           [
               "916.588",
               5,
               -87
           ],
           [
               "916.612",
               5,
               -84
           ],
           [
               "916.636",
               5,
               -82
           ],
           [
               "916.660",
               5,
               -82
           ],
           [
               "916.684",
               5,
               -83
           ],
           [
               "916.708",
               5,
               -83
           ],
           [
               "916.732",
               5,
               -91
           ]
       ],
       "setFreq": 916.66,
       "usedDefault": false,
       "timestamp": "2017-12-29T07:46:28.000Z"
   },
   "uploader": {
       "battery": 78,
       "batteryVoltage": 3958
   },
   "created_at": "2017-12-29T11:12:26.308Z"
   }