Byteflies update, closes #27

[davidverweij]

These changes ensure that the ByteFlies branch follows a similar flow to the Dreem branch, including folder structure, batch-processing by patient+device_id as well as logging.

Partially responds to #28 (at least for testing)

Test

• pull down branch
• run poetry install
• run poetry shell
• run poetry run nox -rs for all tests
• run poetry run consumer to run the inventory for the overall pipeline
• ensure that you have the mongo DB running
• ensure your .envs are correctly set - including ACCESS_AUTHENTICATED_ENDPOINTS=True in .consumer.env
• run python data_transfer/main.py BTF Kiel 0 1 to run the BTF pipeline for today (0) and yesterday (1)

For testing the historical data pipeline, I suggest to add return after logging the data period here, and run:

• python data_transfer/main.py BTF Kiel -1

[davidverweij]

@jawrainey responded to your comments - please have another look.

[jawrainey]

Great work @davidverweij. This works for me locally (also in Docker locally) to get all existing records for Kiel/Munster. Adding the option to get historical data works perfectly too. Super job on adding quite a range of complex testing.

I've two very minor comments, but otherwise it's functionally perfect:

1. As __key_by_value is not used in byteflies.py we could remove it?
2. Are the print statements here intended to exist or should these be log.debug instead?

[davidverweij]

@jawrainey Good catches. I've removed the __key_by_value. The print statements were a sloppy leftover of my testing. I've also re-added the time.sleep(4) as you have for DRM to mitigate the (current) concerns around overloading the inventory api.