Python uploader for real time data to NightScout

[ondrej1024]

I am looking into writing a Python uploader for real time pump and sensor data to NightScout, based on the existing code from this repo.

Unfortunately the NightScout documentation is a bit confusing so I thought I'd ask here. Can you please point me to a good description of the NightScout uploader API?

Thanks

[Pogman]

I don't think there is much. Look through the NS source for clues and maybe the Android uploader will help there too.

[ondrej1024]

I found it. Needed to install an instance of Nightscout. Then the /api-docs/ URL has all the info (see below).

I just don't understand how the access token is passed in the API request? Can you please shed some light on this?

Thanks

[ondrej1024]

Never mind, I worked out the authorization issue.

I am now able to send a single SGV data point to my Nightscout instance. The following curl command returns success. However no data appears in the Nightscout GUI. Any ideas what is going wrong?

curl -X POST "http://10.5.0.7:8080/api/v1/entries.json" -H "accept: */*" -H "api-secret:cc8a9a7c3670b6d3966fff040238f01a99da6b38" -H "Content-Type: application/json" -d " [{\"type\":\"sgv\",\"date\":1573470155,\"sgv\":100}]"

[ondrej1024]

Oh, I just realized that the data record I am sending to the API is actually rejected so that's why it doesn't show up in the GUI. So what could be the reason for the data being rejected?

[ondrej1024]

I found the solution with a comment in the new issue I raised here.

Basically the Nightscout documentation is far from perfect so it is really challenging finding the correct solution. But I guess you guys have gone through all this while implementing the 600SeriesAndroidUploader :smirk:

[steve8x8]

If you find how to remove an entry, please share your insight.

[Pogman]

It's been quite some time since I've worked with the NS API and I've forgotten the details. The android uploader has all this worked out: https://github.com/pazaan/600SeriesAndroidUploader/tree/master/app/src/main/java/info/nightscout/android/upload/nightscout

To remove a item do a DELETE with http query form. /api/v1/treatments.json?find[_id]=12345 /api/v1/treatments.json?find[created_at]=12345

From the JAVA code:

// query based delete @DELETE("/api/v1/treatments.json") Call<ResponseBody> deleteID(@Query("find[created_at]") String date, @Query("find[_id]") String id);

[Pogman]

Break it into chunks. Allow for network issues all along the chain and make sure each part completes successfully. Allow for memory usage on the NS server and the uploader. While processing any incoming bulk of data other operations may be blocked as it processes and writes to the mLab DB. I don't know what the upper limit is here but I would keep bulk uploads to 100 entries.

[steve8x8]

@pogman: Thanks for the advice which was too late already - for now, the DB has survived the flood of data, and it won't happen very soon again.

@ondrej1024: I presume this whole effort will culminate in a sibling of ddguard that uploads directly to a NS instance, making your blynk approach obsolete? I (and some others)'ve been waiting for a non-Java, non-phone 600series uploader - which seems to be within reach now (if only my Python wasn't that rusty...).

To make the device even smaller, a RasPi ZeroW comes to mind - further miniaturization would be limited by (a) the CNL and (b) the power supply. (There's microPython for the ESP8266, ESP32 IoT devices, but one would have to reinvent the USB stack, I'm afraid.)

Do you have an estimate of the power draw of the CNL alone? That is, which amount of battery capacity would be necessary to run it?

[ondrej1024]

@steve8x8 the Nightscout upload feature will only be an option of DD-Guard as I was asked to add that. The Blynk API will remain the main upload path to the Smarphone viewer app because that's what I prefer.

I have already purchased a RPi ZeroW which is on it's way, to run some test with DD-Guard. I hope it can be powered with a power bank together with the CNL but it's too early to say if this is actually working.

[ondrej1024]

Hi all, finally, due to the Corona crisis, I am less involved with my job and I have some time to dedicate to this project. After a few days of intensive work I have now backported some of the improvements that @pazaan and @Pogman implemented in the Android uploader code.

With theses changes, the CNL2.4 Python driver works reliably here for getting live data from the 670G pump with no E86 error, running on a battery powered RPi-Zero.

Next I will try to fix the history download function.

If you are interested, here is the modified Python driver: https://github.com/ondrej1024/ddguard/blob/comms_robustness_backport/cnl24driverlib.py

I also started to write some documentation to better understand how the driver is working: https://github.com/ondrej1024/ddguard/blob/comms_robustness_backport/doc/cnl24-driver-doc.md