search

openmhealth / shimmer

An application for reading health data from third-party APIs.
Apache License 2.0
443 stars 122 forks source link

readme

Shimmer Build Status Join the chat at https://gitter.im/openmhealth/shimmer

Shimmer is an application that makes it easy to pull health data from popular third-party APIs like Runkeeper and Fitbit. It converts that data into an Open mHealth compliant format, letting your application work with clean and clinically meaningful data.

We currently support the following APIs

This README should have everything you need to get started. If you have any questions, feel free to open an issue, email us, post on our form, or visit our website.

Contents

Overview

Shimmer is made up of different components - individual shims, a resource server, and a console - which are each described below.

Shims

A shim is a library that can communicate with a specific third-party API, e.g. Fitbit. It handles the process of authenticating with the API, requesting data from it, and mapping that data into an Open mHealth compliant data format.

A shim generates data points, which are self-contained pieces of data that not only contain the health data of interest, but also include header information such as date of creation, acquisition provenance, and data source. This metadata helps describe the data and where it came from. The library is called a shim because such clean and clinically significant data is not provided natively by the third-party API.

Resource server

The resource server exposes an API to retrieve data points. The server handles API requests by delegating them to the correct shim. As more and more shims are developed and added to the resource server, it becomes capable of providing data points from more and more third-party APIs. The resource server also manages third-party access tokens on behalf of shims.

Console

The console provides a simple web interface that helps users interact with the resource server. It can trigger authentication flows, and request data using date pickers and drop downs.

Installation

There are two ways to install Shimmer.

  1. You can download and run pre-built Docker images.
  2. You can build all the code from source and run it natively or in Docker.

Option 1. Download and run Docker images

If you don't have Docker and Docker Compose, please set them up. (Docker for Mac, Docker for Windows).

Once you're set up, in a terminal

  1. Clone this Git repository.
  2. Download and start the containers using either
    • docker-compose up -d resourceserver
      • to bring up only the resource server
    • docker-compose up -d
      • to bring up the resource server and the console
    • This will download up to 0.5 GB of Docker images if you don't already have them, the bulk of which are the underlying MongoDB, nginx and OpenJDK images.
    • If you want to see logs and keep the containers in the foreground, omit the -d.
  3. It can take up to a minute for the containers to start up. You can check their progress using docker-compose logs if you started with -d.
  4. The console container publishes port 8083 and the resource server container publishes port 8084.
    • The console container proxies all API requests to the resource server container, so you can send API requests to port 8083 or port 8084.
  5. Visit http://<shimmer-host>:8083 in a browser to open the console.

Option 2. Build the code and run it natively or in Docker

If you prefer to build the code yourself,

  1. You must have a Java 8 or higher JDK installed. You can use either OpenJDK or the Oracle JDK.
  2. If you're building the optional console,
    1. You need Node.js.
    2. You need Xcode Command Line Tools if you're on a Mac.
  3. To run the code natively, you need a running MongoDB instance.
  4. To run the code in Docker, you need Docker and Docker Compose.

If you want to build and run the code natively, in a terminal

  1. Clone this Git repository.
  2. Run the ./run-natively.sh script and follow the instructions.
  3. When the script blocks with the message Started Application, the components are running.
    • Press Ctrl-C to stop them.
    • The script creates a WAR file which you can alternatively drop into an application server. This issue has details.
  4. Visit http://<shimmer-host>:8083 in a browser to open the console.

If you want to build and run the code in Docker, in a terminal

  1. Clone this Git repository.
  2. Run the ./run-dockerized.sh script and follow the instructions.
    • The containers should now be running on your Docker host and expose ports 8083 and 8084.
    • It can take up to a minute for the containers to start up.
  3. Visit http://<shimmer-host>:8083 in a browser to open the console.

If you can't run the Bash scripts on your system, open them and take a look at the commands they run. The important commands are marked with a "#CMD" comment.

Registering with third-party APIs

To get data from a third-party API, you need to visit the developer website of that API and register a client application. The registration information that you give to the third-party lets them show relevant information to their end users about your application, and lets them manage other operational concerns like authorization and rate limits.

You will be given a set of client credentials, usually an OAuth client ID and client secret, for each application you register. You may also need to enter a redirect URL, which is the URL a user is sent to after granting your application access to their data.

The following table contains a link to the developer portal of each API and information about redirect URL restrictions. The restrictions can be good to know about during development, but TLS and full URLs should be used during production.

API requires TLS allows non-FQDN hostname allows IP addresses allows localhost requires URL path example
Fitbit1 false true true true true http://localhost:8083/authorize/fitbit/callback
Google Fit false false false true ? http://localhost:8083/authorize/googlefit/callback
iHealth2 ? ? ? ? false http://localhost:8083/authorize/ihealth/callback
Jawbone UP false 3 ? ? ? ? http://localhost:8083/authorize/jawbone/callback
Misfit ? ? ? ? ? http://localhost:8083/authorize/misfit/callback
Moves ? ? ? ? ? http://localhost:8083/authorize/moves/callback
RunKeeper ? ? ? ? ? http://localhost:8083/authorize/runkeeper/callback
Withings ? true ? true ? http://localhost:8083/authorize/withings/callback

1 Fitbit has deprecated OAuth 1.0a authorization in favour of OAuth 2.0. You will need OAuth 2.0 credentials.

2 You'll need to copy the iHealth SC and SV values found via the application management page into the application.yaml or resource-server.env file.

3 The documentation states TLS is required, but authorization does work without it.

If any of the links or fields are incorrect or out of date, please submit an issue to let us know. This table will be fully populated in in the coming days.

Visit the links to register and configure your application for each of the APIs you want to use. Once credentials are obtained for a particular API, you can either set the corresponding values in the application.yaml file and rebuild, or if you're running using Docker, set the corresponding values in the resource-server.env file.

Postman collection

A Postman collection is provided that makes it easy to experiment with Shimmer's API. An environment is also provided that contains settings for making API requests, such as Shimmer's host, port, and request parameters.

Setting up the Postman environment

To set up the Postman environment,

  1. Click the cog wheel in the top right.
  2. Choose Manage Environments.
  3. Click the Import button and choose the file called resources/postman/postman-environment.json from this repo.
  4. Close the Manage Environments modal.
  5. Select the environment you just created from the drop-down in the top right.

Importing the Postman collection

To import the Postman collection,

  1. Click the Import button in the top left.
  2. In the Import File tab, click Choose Files and choose the file called resources/postman/postman-collection.json from this repo.
  3. If you already have a collection with the same name, replace it.

Authorizing access to a third-party user account

The data produced by a third-party API belongs to some user account registered on the third-party system. To allow a shim to read that data, you'll need to initiate an authorization process. This process lets the user account holder explicitly grant the shim access to their data.

Authorize access using Postman

To initiate the authorization process using Postman,

  1. Click the Environment quick look button in the top right of Postman and click Edit to edit the environment.
  2. Set the username value to any unique identifier you'd like to use to identify the user.
  3. Set the shim-key value to one of the keys listed below, e.g. fitbit.
  4. Run the Authorization\Initiate authorization request.
  5. Find the authorizationUrl value in the returned JSON response and load the URL in a browser. You will land on the third-party website where you can login and authorize access to your third-party user account. You should then be automatically redirected back to Shimmer where the OAuth flow will complete.

Authorize access programmatically

To initiate the authorization process programmatically,

  1. Make a GET request to http://<shimmer-host>:8083/authorize/{shimKey}?username={userId}
    • Use port 8084 if you're not running the console container.
    • The shimKey path parameter should be one of the keys listed below, e.g. fitbit.
    • The username query parameter can be set to any unique identifier you'd like to use to identify the user.
  2. Find the authorizationUrl value in the returned JSON response and redirect your user to this URL. Your user will land on the third-party website where they can login and authorize access to their third-party user account.
  3. Once authorized, they will be redirected to http://<<shimmer-host>:8083/authorize/{shimKey}/callback.

Authorize access from the console

To initiate the authorization process from the console,

  1. Type in an arbitrary user handle. This handle can be anything, it's just your way of referring to a user.
  2. Press Find and the console will show you a Connect button for each API with configured authentication credentials.
  3. Click Connect and a pop-up will open.
  4. Follow the authorization prompts.
  5. After following the prompts, you should see an authorization successful response in the pop-up.
  6. The pop-up will then automatically close.

Reading data

A shim can produce JSON data that is either normalized to Open mHealth schemas or in the raw format produced by the third-party API. Raw data is passed through from the third-party API. Normalized data conforms to Open mHealth schemas.

The following is an example of a normalized step count data point retrieved from Jawbone:

{
    "header": {
        "id": "243c773b-8936-407e-9c23-270d0ea49cc4",
        "creation_date_time": "2015-09-10T12:43:39.138-06:00",
        "acquisition_provenance": {
            "source_name": "Jawbone UP API",
            "modality": "sensed",
            "source_updated_date_time": "2015-09-10T18:43:39Z"
        },
        "schema_id": {
            "namespace": "omh",
            "name": "step-count",
            "version": "1.0"
        }
    },
    "body": {
        "effective_time_frame": {
            "time_interval": {
                "start_date_time": "2015-08-06T05:11:09-07:00",
                "end_date_time": "2015-08-06T23:00:36-06:00"
            }
        },
        "step_count": 7939
    }
}

Read data using Postman

To pull data from a third-party API using Postman,

  1. If you need to modify the environment, click the Environment quick look button in the top right of Postman and click Edit to edit the environment.
  2. Fill in the date range you're interested in by setting the start-date and end-date values.
  3. Set the normalized value to true for data that has been converted to an Open mHealth compliant format, or false for raw data.
  4. Run the Data points request you're interested in.

Most requests also have sample responses you can look at by clicking the Examples button in Postman. Please let us know if any examples you need are missing, or provide pull requests to contribute.

Read data programmatically

To pull data from a third-party API programmatically, make requests in the format

http://<<shimmer-host>>:8083/data/{shimKey}/{endpoint}?username={userId}&dateStart=yyyy-MM-dd&dateEnd=yyyy-MM-dd&normalize={true|false}

Use port 8084 if you're not running the console container.

The URL can be broken down as follows

N.B. This API may change significantly in the future to provide greater consistency across Open mHealth applications and to improve expressivity and ease of use. The data points it returns will not be affected, only the URLs used to request data and perhaps some book-keeping information at the top level of the response.

Read data using the console

To pull data from a third-party API using the console,

  1. Click the name of the connected third-party API.
  2. Fill in the date range you're interested in.
  3. Press the Raw button for raw data, or the Normalized button for data that has been converted to an Open mHealth compliant data format.

Supported APIs and endpoints

The following is a table of the currently supported shims, their endpoints, the Open mHealth compliant data produced, and the corresponding mapper. The values in the shim key and endpoint columns are the values for the parameters of the same names used in programmatic access of the API.

The currently supported shims are:

shim key endpoint OmH data produced by endpoint mapper
fitbit1 body_mass_index omh:body-mass-index:2.0 FitbitBodyMassIndexDataPointMapper
fitbit1 body_weight omh:body-weight:1.0 FitbitBodyWeightDataPointMapper
fitbit1 heart_rate (intraday) omh:heart-rate:1.1 FitbitIntradayHeartRateDataPointMapper
fitbit1 physical_activity omh:physical-activity:1.2 FitbitPhysicalActivityDataPointMapper
fitbit1 sleep_duration omh:sleep-duration:2.0 FitbitSleepDurationDataPointMapper
fitbit1 sleep_episode omh:sleep-episode:1.0 FitbitSleepEpisodeDataPointMapper
fitbit1 step_count2 omh:step-count:2.0 FitbitStepCountDataPointMapper
fitbit1 step_count (intraday)2 omh:step-count:2.0 FitbitIntradayStepCountDataPointMapper
googlefit body_height omh:body-height:1.0 GoogleFitBodyHeightDataPointMapper
googlefit body_weight omh:body-weight:1.0 GoogleFitBodyWeightDataPointMapper
googlefit calories_burned omh:calories-burned:2.0 GoogleFitCaloriesBurnedDataPointMapper
googlefit geoposition omh:geoposition:1.0 GoogleFitGeopositionDataPointMapper
googlefit heart_rate omh:heart-rate:1.1 GoogleFitHeartRateDataPointMapper
googlefit physical_activity omh:physical-activity:1.2 GoogleFitPhysicalActivityDataPointMapper
googlefit speed omh:speed:1.0 GoogleFitSpeedDataPointMapper
googlefit step_count omh:step-count:2.0 GoogleFitStepCountDataPointMapper
ihealth blood_glucose omh:blood-glucose:1.0 IHealthBloodGlucoseDataPointMapper
ihealth blood_pressure omh:blood-pressure:1.0 IHealthBloodPressureDataPointMapper
ihealth body_mass_index omh:body-mass-index:1.0 IHealthBodyMassIndexDataPointMapper
ihealth body_weight omh:body-weight:1.0 IHealthBodyWeightDataPointMapper
ihealth heart_rate omh:heart-rate:1.1 IHealthHeartRateDataPointMapper
ihealth physical_activity omh:physical-activity:1.2 IHealthPhysicalActivityDataPointMapper
ihealth sleep_duration omh:sleep-duration:2.0 IHealthSleepDurationDataPointMapper
ihealth step_count omh:step-count:2.0 IHealthStepCountDataPointMapper
jawbone body_mass_index omh:body-mass-index:1.0 JawboneBodyMassIndexDataPointMapper
jawbone body_weight omh:body-weight:1.0 JawboneBodyWeightDataPointMapper
jawbone heart_rate3 omh:heart-rate:1.1 JawboneHeartRateDataPointMapper
jawbone physical_activity omh:physical-activity:1.2 JawbonePhysicalActivityDataPointMapper
jawbone sleep_duration omh:sleep-duration:1.0 JawboneSleepDurationDataPointMapper
jawbone step_count omh:step-count:1.0 JawboneStepCountDataPointMapper
misfit physical_activity omh:physical-activity:1.2 MisfitPhysicalActivityDataPointMapper
misfit step_count omh:step-count21.0 MisfitStepCountDataPointMapper
misfit sleep_duration omh:sleep-duration:2.0 MisfitSleepDurationDataPointMapper
misfit sleep_episode omh:sleep-episode:1.0 MisfitSleepEpisodeDataPointMapper
moves4 physical_activity omh:physical-activity:1.2 MovesPhysicalActivityDataPointMapper
moves4 step_count omh:step-count:1.0 MovesStepCountDataPointMapper
runkeeper calories_burned omh:calories-burned:2.0 RunkeeperCaloriesBurnedDataPointMapper
runkeeper physical_activity omh:physical-activity:1.2 RunkeeperPhysicalActivityDataPointMapper
withings blood_pressure omh:blood-pressure:1.0 WithingsBloodPressureDataPointMapper
withings body_height omh:body-height:1.0 WithingsBodyHeightDataPointMapper
withings body_weight omh:body-weight:1.0 WithingsBodyWeightDataPointMapper
withings calories_burned5 omh:calories-burned:2.0 WithingsDailyCaloriesBurnedDataPointMapper
withings calories_burned (intraday)5 omh:calories-burned:2.0 WithingsIntradayCaloriesBurnedDataPointMapper
withings body_temperature omh:body_temperature:1.0 WithingsBodyTemperatureDataPointMapper
withings heart_rate omh:heart-rate:1.1 WithingsHeartRateDataPointMapper
withings sleep_duration6 omh:sleep-duration:2.0 WithingsSleepDurationDataPointMapper
withings sleep_episode6 omh:sleep-episode:2.0 WithingsSleepEpisodeDataPointMapper
withings step_count5 omh:step-count:2.0 WithingsDailyStepCountDataPointMapper
withings step_count (intraday)5 omh:step-count:2.0 WithingsIntradayStepCountBurnedDataPointMapper

1 The Fitbit API doesn't provide time zone information for the data points it returns. Furthermore, it is not possible to infer the time zone from any of the information provided. Because Open mHealth schemas require timestamps to have a time zone, we need to assign a time zone to timestamps. We set the time zone of all timestamps to UTC for consistency, even if the data may not have occurred in that time zone. This means that unless the event actually occurred in UTC, the timestamps will contain an incorrect time zone. Please consider this when working with data normalized into OmH schemas that are retrieved from the Fitbit shim. We will fix this as soon as Fitbit makes changes to their API to provide time zone information.

2 The configuration file controls whether to serve Fitbit intraday or summary data and at what granularity (see application.yaml or resource-server.env for details). Intraday activity requests are limited to 24 hours worth of data per request. Fitbit must enable intraday access explicitly for your application (click the endpoint link for details). Attempting to generate normalized data with the intraday access property set to true, but when your API credentials have not been granted intraday access, will result in an error.

3 The heart rate mapper has not been tested on real data from Jawbone devices. They have been tested on example data provided in Jawbone API documentation. Please help us out by testing Shimmer with real-world data of one of these types from a Jawbone device and letting us know whether or not it works correctly.

4 Moves time zone handling needs to be tested further, as it's not clear if the time zone assumptions in the mappers are correct.

5 The Withings configuration controls whether to serve intraday or summary data (see application.yaml or resource-server.env for details). Intraday activity requests are limited to 24 hours worth of data per request.

6 Sleep data has not been tested using real data directly from a device. It has been tested with example data provided in the Withings API documentation. Please help us out by testing real-world Withings sleep data with Shimmer and letting us know whether or not it works correctly.

Contributing

The list of supported third-party APIs will grow over time as more shims are added. If you'd like to contribute a shim to work with your API or a third-party API, or contribute any other code,

  1. Open an issue to let us know what you're going to work on.
    1. This lets us give you feedback early and lets us put you in touch with people who can help.
  2. Fork this repository.
  3. Create your feature branch from the develop branch.
  4. Commit and push your changes to your fork.
  5. Create a pull request.