Tidepool Integration

[jamorham]

xDrip should have the ability to upload data to Tidepool.

[doc77]

That would be awesome! They brag on being very flexible and open source, so they should provide an easy to implement pathway to integration.

[jwoglom]

Appears as though tidepool's public API documentation is a bit lacking.. reaching out to them to see if they have an updated version of the info on this page. (the linked google doc is unavailable) https://developer.tidepool.org/tidepool-api/index/

[jweismann]

@jamorham - Sounds great (as always). I have one design question related to sync follower setups and uploads to tidepool and other platforms. I have noticed that with NS rest api uploads I need to upload from followers too in order not to have a complete representation in NS. If I only upload from one phone in the sync-follower setup then I only get the data that is entered on this particular phone and not the data coming from the fellow followers. Is this intentional by design (or am I simply using it wrong) or would it be possible to have either the master or one of the followers to upload on behalf on the sync group?

[dmelliott20]

This would be great

[n0rbertg]

+1

[yeagerm]

Yes, I agree, that would be helpful

[stackenblochen]

Sounds great+1

[kskandispersonal]

Agree, this would be awesome.

[bassettb]

I have a proof-of-concept Tidepool uploader working that takes an xDrip+ exported file (SiDiary format) and uploads the data to Tidepool. It is using their new, and yet unreleased API. There's one messy workaround for uploading carbs. The reason an updated doc is not published is because the new API is not yet done and released. I think we will have to wait for its release to do any real work with it.

[NRiggall]

Hello!

@bassettb is correct - Tidepool is still working toward the release of our public open APIs which will really open the floodgates for future integrations. Although there are a few exceptions - we have not given many access to info about the open API as so much is in flux and we do not want to waste anyone's efforts by letting them build on the API and then pulling the rug from beneath them with drastic changes.

I am a big fan of all of the work of those who have worked on xDrip+, Nightscout, etc and I cannot wait to see this type of integration exist.

Thank you all and thank you @jamorham !

[jamorham]

So I have a working implementation for xDrip+ written now (many thanks to @bassettb for the proof-of-concept c# example tool and Tidepool for access to the api)

I've implemented it to be as modular and adaptable as possible in terms of the data structures, rest-api endpoints and the message sequence, bearing in mind the expectation that the api may be subject to change. I am fairly confident it will be possible to modify this very quickly as required.

It supports CGM data from all collectors, manual and automated synced blood test data, treatment items entered in xDrip or via Nightscout or from any followers or pump synchronization. It uses the data pulled from AAPS to upload the dynamic basal rate information.

To avoid problems with duplicated data I went with a completely linear upload strategy from oldest to newest never repeating the same uploaded time segment. It seems to take about 1 second to upload an entire days worth of data.

Overall it is looking good and a very worthy addition to xDrip in my opinion as this is something I would rate as an urgent priority for the community.

As far as I understand it, the next steps would need to be Tidepool agreeing that it would be possible for the api to be utilized by xDrip users. I will contact Tidepool to discuss this and hopefully we can get interested people testing it soon.

[doc77]

@JamOrHam This is great news! Do you have a time frame for its inclusion into xDrip+?

Many thanks to @bassettb for his hard work and to you for all you do for us xDrip+ users.

[kskandispersonal]

Thank you!! Looking forward to using this feature. Question... do you think you could also go the other way? If a user uploads their pump data to Tidepool, can you retrieve it into xDrip+ and NS?

[jamorham]

@doc77 well the code is ready as far as I'm concerned but Tidepool need to be willing to receive data from xDrip. I would expect first steps would be a limited trial of nightly+engineering mode users perhaps.

@kskandispersonal I haven't yet been able to retrieve datasets from tidepool, but if we could do that then yes definitely we could sync in both directions.

[gregorybel]

what are the benefits to use tidepool? as far as I have understood it seems to be a NS like?

[jamorham]

Discussions with Tidepool this looks like it is going to happen. No ETA yet though.

[doc77]

@jamorham Awesome! Thank you very much. I think this will be a great enhancement to the already indispensable app that you have developed.

[yepee1]

@jamorham Thank you ! 👍

[cramis1]

Really looking forward to this! Williing to beta test if needed

[9zr5EHJA]

Thank you to all involved - I'm looking forward to this integration, too.

[jamorham]

There is some movement on this so hopefully it wont be too much longer till we can start the first round of testing with nightly users.

[doc77]

Will be happy to be a beta tester.

[n0rbertg]

Just wondering what's the status with this. Is there any update from Tidepool?

[pazaan]

I really appreciate your interest and patience on this, everyone! We (Tidepool) were in the middle of working on a new upload delivery mechanism that is better suited to real-time applications like mobile uploads (obviously including xDrip+). This PR makes use of the new, continuous mechanism. This also enables the xDrip+/Tidepool integration to be widely adopted without causing any negative impact on Tidepool's servers. Thanks for all the help @jamorham!

[doc77]

Integration with Tidepool is a major step, and adds a new dimension to xDrip. Thank you @jamorham for doing this!

[jamorham]

This feature is in the 28th Nov Nightly for early testing. To use it you need to register at: https://int-app.tidepool.org/ and then input your credentials to the Tidepool cloud upload section in xDrip settings.

Data should be uploaded every 20 minutes. There is currently very little feedback in the user interface as to whether it is working or not so you would need to refresh on the website to check the data.

This is still early stages and requires testing and feedback from early adopters!

[doc77]

I had to sign up for a new Tidepool account at the link. I entered in my original tidepool credentials and set up a new int-app account. Entered the credentials into xDrip+. The "Test Tidepool Login" seems to be non-funcional as it did nothing when I clicked it.

Its been about an hour and a half and data has not been uploaded to Tidepool. Log shows multiple login requests that failed. Got the message that "login failed: java.net.UnknownHostException: Unable to resove host "int-api.tidepool.org" No address associated with hostname"

[doc77]

Should the hostname be "int-app.tidepool.com"?

[doc77]

Update: Its working! Tidepool data chunks are uploading about every 10 minutes. Started several weeks before and is almost caught up to today. Data shows in Tidepool.

Thank you for this @ jamorham!

[jamorham]

Yes, so I should also have mentioned it will try to upload data going back 2 months in chunks of 2-3 days until it has caught up. At every stage we are doing everything possible to ensure xDrip never causes Tidepool any load related problems and is always well behaved and polite.

edit: I've been informed that we probably do not need to be this cautious and could upload much bigger chunks and tidepool servers wont blink, so I will work on testing that and updating the code. Also compression isn't currently enabled but I would plan to switch that on too.

[kskandispersonal]

The "Test Tidepool Login" generates "Login was not successful: 401 Unauthorized" even though I am using my existing Tidepool credentials which work on the Tidepool site. Does it need to be a new account?

[jamorham]

@kskandispersonal Yes, new login is required as this is the int site. See https://github.com/NightscoutFoundation/xDrip/issues/581#issuecomment-442517992

[kskandispersonal]

@jamorham Thanks, just signed up w/ new account:-) And now uploading! Thank you!!!!

[doc77]

I had to create a new account (with the same credentials) on the web site @jamorham refers to. It took about 2 hours for the uploads to start, and about four hours to load about 2 months worth of data. And the data was available on both sides, the original Tidepool and the int-app.tidepool.org.

[kskandispersonal]

@doc77 Thanks, xDrip+ Event Log is showing the TidpoolUploadChunk so it should be doing the upload. I don't see anything on the Tidepool site yet.

Are you saying that you can see your data on both the original site, https://app.tidepool.org, and the one referenced above, https://int-app.tidepool.org/? That would be great. I already have data on the original site such as insulin dosage that are not on nightscout and xdrip so I'd like to keep that data.

[pazaan]

Are you saying that you can see your data on both the original site, https://app.tidepool.org, and the one referenced above, https://int-app.tidepool.org/?

@kskandispersonal No, you will only see data on https://int-app.tidepool.org during this testing period.

[kskandispersonal]

@pazaan Okay, Thank you!! I am seeing my BGs updated now on the new site:)

[pazaan]

@kskandispersonal once testing is complete, we will point xDrip+ to Tidepool's production server. But we do want to take the time to test properly to ensure that the data uploaded is accurate. Especially since a popular use-case for this integration is sharing data with clinicians.

[doc77]

@kskadispersonal Arrgh! I was mistaken. I can't see the newly updated data on the original app.tidepool.org site.

[kskandispersonal]

@pazaan Understood. Yes, indeed, testing is important! I just uploaded my omnipod data to the test site:)

@doc77 No problem:) It will be soon!

[kskandispersonal]

@pazaan Ah, okay, the uploader, of course, uploads the pump data to the production site!!

[pazaan]

@kskandispersonal Right. The Tidepool Uploader can upload to int as well if you want, but if you do that, you run the risk of duplicates (if there are duplicate data points, like boluses or wizard entries)

[kskandispersonal]

@pazaan How would I configure the Uploader to upload to the int site? I can of course wait 'til the test is complete, but I wouldn't mind testing the complete view of my treatments:)

[pazaan]

@kskandispersonal if you right-click in the Tidepool Uploader app's login page (log out if you're already logged in), you will get a context menu for "Change Server" that lets you change to "Integration".

[kskandispersonal]

@pazaan I'm not getting the context menu. Should I right click on a specific element? I logged out of the uploader and it is just showing the credential textboxes and the Login button. v0.312.0 showing in lower left corner.

[kskandispersonal]

@pazaan I reverted back to Uploader v2.8.6 and now it is showing the change server context menu. Thank you! I will try it.

[pazaan]

@kskandispersonal I was about to say - looks like you're running a really old uploader!

[kskandispersonal]

@pazaan Nice, uploaded all of the omnipod data:)

[jamorham]

29th Nov nightly contains updates/improvements to Tidepool support. If you are testing, please update to this version. Improvements in user feedback, for example you can now see the status in System Status Uploader tab and tapping Last Poll can force an immediate upload.

[doc77]

@jamorham. It seems that a treatment entered in the past, after that chunk upload, is not included in subsequent uploads, so it never makes it to Tidepool.