New APIs - platform improvement - extremely useful overlooked datapoints

[biblicabeebli]

There are several new datasets and data APIs that have been proposed and discussed, this issue serves as the final engineering-level issue required to coordinate us to get these built. These discussions are coming out of meetings, emails, and over in the private Onnela Lab discussions repo (so if anyone external is watching and has any ideas feel free to discuss in comments here.)

Summary of current APIs (before making additions):

Data Access APIs - all of these APIs that use the Data Access API Credentials

• get-data/v1 The Main Data Access API - this is the ~direct data sourced from the apps. It is files, they are downloaded as csvs in a(n uncompressed) zip file. API requests can be extensively filtered, but are limited to one study at a time.
• get-studies/v1 Provides a JSON list of study ids and their names. (provides only the studies the user has access to)
• get-users/v1 Provides a JSON list of participant ids in the queried for study.
• get-interventions/v1 Provides a JSON list of the data about all interventions (including history) for all participants in a study.

Tableau API - this API uses the Tableau credentials, which look-like-but-are-not the Data Access Credentials

• api/v0/studies/<study_id>/summary-statistics/daily/wdc This is the boutique Tableau login page known in proprietary Tableau jargon as the Web Data Connector
• api/v0/studies/<study_id>/summary-statistics/daily This is the API endpoint, it provides paginated data of the Daily summaries, sourced from raw uploaded data for quantity statistics, and derived Forest-generated statistics. API requests can be extensively filtered and the list of available fields grows as we add more data. (and I think like get-data it requires the study id but its hard to tell from the code)

Pseudo APIs - we have some items that are accessible from the website but may not have api endpoints

• The Forest Task Page has a downloadable zip of ALL output data from a single Forest task.
• ~The Forest Task Page has a link that effectively automatically provides the (current! not historical!) output of the Data Access API call that was made to download data that the Forest Task ran on.~
  • We don't need an api endpoint for this, it is very niche and really only for debugging.
• The Edit Study Page has a button that exports a JSON blob of the current study-settings data for the study - which includes as a subcomponent the content of the current surveys.
  • ~The Edit Study Page has a button that downloads the intervention history~ it's the same as the get-interventions/v1 api endpoint.
  • The Edit Study Page has a button that exports a JSON blob of survey and survey history data.
• As of March 14th there is a API endpoint for this one. It's not on production servers yet.
  • I guess the Participant Push Notification History Page is also a pseudo-api - it is the only place where one can view a history of keepalive notifications.

Summary of APIs (or pseudo APIs) to add:

• A Data Access API endpoint that provides the same data set that is visible on the View Study page, e.g. the participant metadata + custom fields defined by the study.
  • ~a key extra metadata data-point that would be a great add is the original registration date.~
  • ~additional trivial metadata like the last upload attempt, e.g. the contents of the Participant database model.~
  • Fully implemented as a button on the view study page and is on our staging server. Decent test coverage, still requires some user feedback.
  • ~TODO: implement as a new api endpoint - json or csv...~
  • new api endpoint is implemented.
• ~finish the survey-download-api branch.~
  • done!
• Heartbeat Data - heartbeat is a new data stream for app debugging and platform enhancement, it is sourced from the app and serves as a dedicated periodic signal that the app has checked in.
  • ~This feature on the website is still gated by the participant experiment button (which must be enabled manually on servers). We are moving to generally enable this, but it is now present on version 2.4.6-aka-2.4.13 of the iOS app.~
  • Feature is fully live.
  • Endpoint is implemented.
• ~Upload/Upload timeliness information - the time, type, size, and name (which always has a file creation timestamp so we can calculate "upload lag")~
  • We have been storing this data for ages, it can be turned into a new API endpoint.
  • implemented!
  • made some adjustments (done).
• A button at the top of the Forest Tasks page that downloads a single CSV with all summary statistics within in a study for all participant (see sample CSV)
  • implemented v1, on staging
  • minor refinements, tests
  • current implementation also adds a Global total row at the end that is not really valid csv format.
  • reference file: ~data_volume.csv~ NOPE WRONG FILE - it is correct now on this issue: https://github.com/onnela-lab/beiwe-backend/issues/333

Related Items (this list is kept updated)

Where do we host easy access to these APIs? Mano? Probably Mano.

• Update Mano to have access to all of these.
  • ~except tableau because tableau uses tableau api keys, not Data Access api keys~ (don't know what this sentence meant?)
  • ~so we need to implement https://github.com/onnela-lab/beiwe-backend/issues/365~
  • it is implemented
• https://github.com/onnela-lab/beiwe-backend/issues/333
  • ~partially implemented per participant summary statistics button on task log page~
  • ~issue requires discussion~ Issue has been discussed and rebuilt, need feedback on output.
• ~https://github.com/onnela-lab/beiwe-discussions/issues/154~ https://github.com/onnela-lab/beiwe-backend/issues/393
  • there is now a button on the view study page to download the participant table (plus a bunch of extra data points)
  • awaiting feedback.
• ~https://github.com/onnela-lab/beiwe-backend/issues/366~
  • implemented version history as an endpoint

[biblicabeebli]

• https://github.com/onnela-lab/beiwe-backend/issues/333 is related

[biblicabeebli]

Current plan:

do this one

• A Data Access API endpoint that provides the same data set that is visible on the View Study page, e.g. the participant metadata + custom fields defined by the study.

then this one

• Upload/Upload timeliness information - the time, type, size, and name (which always has a file creation timestamp so we can calculate "upload lag")

and then do this one

• A single button at the top of the Forest Tasks page that downloads a single CSV with all summary statistics within in a study for all participant (see sample CSV)

[biblicabeebli]

2 more over in

• https://github.com/onnela-lab/beiwe-backend/issues/366

implemented

[biblicabeebli]

A list of all data api endpoints after finished everything I can find across all issues. There may be additional endpoints and/or refinements. This is not full documentation but it is a start.

~TODO: Document!~ Documented! (see following post for link)

• get-data/v1 - the normal data download endpoint.
• get-studies/v1 - provides a researcher with the list of studies they have access to. Output is a json list.
• ~get-users/v1~ - Deprecated, use get-participant-ids/v1 instead.
• get-participant-ids/v1 - Provides a list of participants on a study. Output is a json list.
• get-interventions/v1 - downloads the Intevervention information for a study. Output is complex json.
• get-survey-history/v1 - downloads all historical changes ever made to surveys. Output is complex json.
• get-participant-data-info/v1 - downloads the total data quantities on each data stream for participants in a study, Output is complex json.
• get-participant-upload-history/v1 - downloads the history of a single participant's uploaded files with timestamps. Output is a json list ordered by timestamp.
• get-participant-heartbeat-history/v1 - downloads the history of when a participant's device sends pings to the server. Output is a json list ordered by timestamp.
• get-participant-version-history/v1 - downloads the version history (app version and os version) of any time these values changed for a participant. Output is a json list ordered by timestamp.
• get-participant-table-data/v1 - downloads an extended version of the table as displayed on the View Study page. Has multiple formats - Json dictionary, json table, and csv format options. (Output is same as the button on the download page.)
• get-summary-statistics/v1 - Is a mirror of the Tableau Summary Statistics endpoint but conforms to the data access api expectations on error behavior. Supports the same list of data filters. Output is a json list of dictionaries.
• get-participant-device-status-history/v1 - Downloads any saved comprehensive participant tracking and debugging data, which is enabled through the Experiments page, accessible by site admins on the participant page. This data is subject to change, it differs depending on app version and whether it is ios or android. WARNING: may be enormous.

[biblicabeebli]

• Bulk backend code work is done, now I need feedback.
  • the download link to a comprehensive csv version of all summary statistics data has been rebuilt, I there was communication glitch.
• Need to update Mano to use/surface these endpoints.
• ~Will first provide a script that can be used to hit any endpoint.~
• A script that can be used to hit any endpoint can be found here: https://github.com/onnela-lab/beiwe-backend/blob/researcher-apis/data_access_api_reference/other_api_reference_script.py
  • Script contains comprehensive documentation of each endpoint. Please let me know if there are details I have forgotten to include.
  • (Please report any bugs in the script, I made some edits and caught a dumb bug in my edits.)

[biblicabeebli]

Update: We are at Done v2 on The Done Scale.

[hydawo]

@biblicabeebli

Let's include "1st registration date/time" as an additional output field in this API get-participant-table-data/v1 - downloads an extended version of the table as displayed on the View Study page. Has multiple formats - Json dictionary, json table, and csv format options. (Output is same as the button on the download page.)

[MMel099]

Just thought of a useful extension for get-users. Two subqueries: get-users-no-data and get-users-data. Think this would be helpful for general study monitoring and to avoid Forest bugs like "no chunked data ..."

[biblicabeebli]

@hydawo @MMel099 I have added the first participant registration field to the participant table CSV and API

(The first registration value is not incorporated as an intervention Thing at this time, that's not part of this work.)

[biblicabeebli]

Just thought of a useful extension for get-users. Two subqueries: get-users-no-data and get-users-data. Think this would be helpful for general study monitoring and to avoid Forest bugs like "no chunked data ..."

I understand the idea and the need, I disagree that get-users, is the appropriate place for that.

• get-users is misnamed, for that reason alone I want to deprecate it.
• get-users should probably get replaced with a get-participant-data-info endpoint that provides the data quantity metrics (I already have logic for this ... because I misinterpreted a different requirement, lol).
• we could also do with a slightly different endpoint to make running a "sync data" command through Mano.
• high-level - allocating a whole api endpoint or parameter like that for an extremely specific datapoint is not a great Design Pattern. Apis are intended to be hit by code, code can work out the answer from data. Different story if we want to include that datapoint on a webpage.

[biblicabeebli]

• because it was super easy I have implemented a get-participant-data-info/v1
• I have deprecated get-users/v1 and added a get-participant-ids/v1 (both currently work, get-users will be removed eventually)

update: reference script with documentation and the listing in the comment above have been updated.

[MMel099]

Just wanted to add some more details about my comments on the /get-interventions endpoint testing. I used the script available here to do the testing.

Original comment: "Returns an empty json - tested on 2 staging studies with confirmed interventions" Study: kt3TqoCqJzGaEebqxkwZIXrQ

Study: GGcUUuyFVGvPwogTsGs4CeNR

[biblicabeebli]

Ohkhay why won't this load on my phone....

[biblicabeebli]

Can you compare your output with The output of the download interventions button in studies settings? (Sometimes I get page names wrong for complex and tedious reasons, and I'm on my phone at the moment. I'm not sure if this is in study settings or somewhere else)

[MMel099]

Thanks Eli, and no urgency on this, just wanted to document! I'll take another look on my end - It may very well be my error.

[jprince127]

Hello!

I wanted to update on something I found when testing the /get-participant-version-history/v1 endpoint of the API. I used the same script as Max above.

On my beta study on the staging server, I found that there are duplicated outputs when I call the API endpoint. I have never unregistered then reregistered my device to this study, so am unsure why this may be.

Server: staging.beiwe.org Name: Jenny_Prince_Test_Study_11.30.23 Study ID: gb0AnPbXW8GswPz767ZgMu0r Participant ID: 966u1cd7

The output is copy-pasted below:

Starting request at 2024-08-07 13:51:51.875640 Request completed at 2024-08-07T13:51:52.334537 duration: 0.458897 seconds http status code: 200 Testing whether it is valid json... JSON successfully loaded into variable json_response [{'app_version_code': '2.5', 'app_version_name': '2024.21', 'os_version': '17.3.1'}, {'app_version_code': '2.5', 'app_version_name': '2024.21', 'os_version': '17.3.1'}, {'app_version_code': '2.5.1', 'app_version_name': '2024.23', 'os_version': '17.3.1'}, {'app_version_code': '2.5.1', 'app_version_name': '2024.23', 'os_version': '17.3.1'}]

[biblicabeebli]

@jprince127 prettified:

[
    {
        "app_version_code": "2.5",
        "app_version_name": "2024.21",
        "os_version": "17.3.1"
    },
    {
        "app_version_code": "2.5",
        "app_version_name": "2024.21",
        "os_version": "17.3.1"
    },
    {
        "app_version_code": "2.5.1",
        "app_version_name": "2024.23",
        "os_version": "17.3.1"
    },
    {
        "app_version_code": "2.5.1",
        "app_version_name": "2024.23",
        "os_version": "17.3.1"
    }
]

OH WOOPS those should Definitiely have timestamps. some time passes Ok I have pushed an update to staging that makes that include timestamps. Please hit the endpoint again and post that output please.

One of these is happening:

• a bug where duplicate entries were getting created in the app version history - I did identify such a bug, I am struggling to recall details and find the bug report - it may be limited to early versions of the implementing code.
• this is a race condition - app hits two endpoints very very rapidly, both instances have code that checks whether reported version numbers have changed but neither has updated the database yet, then both updates get pushed to the database.
  • I think I identified this as the issue..... if so then the timestamps will make it obvious, and it is a bug that is too hard to fix and too minimal for us to care. It should instead be documented over in the script that the endpoint can have duplicates with very close or even identical (they are truncated) timestamps.

So post that output and if its only in ~march? it's fine, if it is very occasional but not limited to march then we update the docs.