Integrate UCAM database API

davidverweij commented 3 years ago

Closes #45

Ps. I had to squash my commits as rebasing gave some headaches

This branch introduces the integration of the UCAM api based on a test API endpoint provided to us. Changes include:

a new consumer endpoint called ucam including 6 sub-endpoints
a different schema for ucam 'records'. Before, these were based on rows in a .csv - now these are based on Patients or Devices, with a sub-list or linked Devices or Patients (Patient-Devices and Device-Patients) respectively.
the ucam endpoints implements authentication similar to the dmpy by reading and writing into the environment. This allows retrieval of auth across pipelines.
an added test-suite to test the consumer implementation of this endpoint as well as the data_transfer implementation. The first focuses on the API, the latter mainly focuses on the additional methods in data_transfer that have some logic to determine patients or IDs based on wear times.
- these test includes integration test, marked with pytest.mark.live - and are excluded from any standard nox session. Instead, you can run these manually: pytest -vm live

Testing

git pull
git checkout ucam_api_2
poetry install
poetry shell
poetry run consumer

In your browser, navigate to

# in another terminal instance
poetry run nox -r
pytest -vm live

# remove the upload step from shared jobs
python data_transfer/main.py BTF Kiel 0 1  # increase last number if no results

davidverweij commented 3 years ago

Added a statement to ignore any UCAM records if they contain deviations. This has to be temporarily used, as currently (once live) UCAM data contains wrongly associated device IDs; since the correct IDs were not available for selection. Running the pipeline where these mis-links are present will cause wrongfully linked data to wrong patients.

Test

Run pytest -vsk "test_get_patient_by_period_date_ignore_deviations" or poetry run nox -rs tests

davidverweij commented 3 years ago

@jawrainey, I've implemented almost all of your feedback, but have had to push a few improvements into issues in order to keep this PR manageable and allow myself to work on other aspects of the pipeline (and our deadlines as is; in particular the refactoring of the tests and additional test structure.

I'm happy to improve and work on those in the future.

davidverweij commented 3 years ago

@jawrainey, UCAM has now added the ByteFlies dots to the payload - which are joined for each device and only in the GET /patients/ endpoint. See the mock data for how that looks like.

I added a (temporary) endpoint GET /btf/ that (under the hood) queries the GET /patients/ endpoint, and iteratively converts any present BTF dots into a list of devices - to be returned as a payload identical to what we expect from GET /devices. It is then serialised as normal.

In the data_transfer, I added one methods to query this endpoint (which caches the result), and another one to do a lookup in that result (to get a single dot).

I've tested locally with python data_transfer/main.py BTF Newcastle 6 6 which resulted in 67 records created and 53 NOT this session. Given these were recent I have higher hopes for more historical data! I could check that a record that was being downloaded was not based on inventory and not found in the ucam /devices endpoint, but was found thanks to the /btf custom endpoint.

Test

pull down changes
run poetry run nox -rs tests

Test btf_dot code with live API

assuming you have credentials for UCAM set up in your env
run poetry run consumer
In _tests/data_transfer_tests/services/testucam.py add the code below, and
replace the key and expected values with existing ID's (see MSTeams for a suggestion)
run pytest -vsk "test_LIVE_get_btfdots"
```
import pytest
```

@pytest.mark.live @pytest.mark.parametrize( "key, expected", [("XXX-XXXXXX", "XXX-XXXXXX"), ("YYY-YYYYYY", "YYY-YYYYYY")] ) def test_LIVE_get_btfdots(key: str, expected: str) -> None: result = ucam.get_one_btf_dot(key) assert result[0].device_id == expected print(result)



## Thoughts?
Let me know what you think and then I feel confident to merge the changes and run the pipeline against - say - Muenster first.

_Note that any deviation is still ignored as I spotted some anomalies. I can ask all clinicians to report back to us when they finished QA so that we may run the pipeline again._

jawrainey commented 3 years ago

Great work @davidverweij 🚀 As you have verified this PR through deploying a dev pipeline for BTF, which I also checked on the DMP and can confirm records are as expected. I have also verified the PR through testing locally and deploying a dev pipeline for DRM, which required merging master given my recent change due to a conflict. This also required building new images to test the deployment as master had a few changes.

ideafast / middleware-services