Initial Data protocol for Byteflies

[davidverweij]

Similar to Dreem (#14) and VTT (#21), the data transfer protocol for ByteFlies needs to be developed. Initial work has been done in the byteflies branch.

Initial tasks to get this up and running include:

• [x] branch into byteflies2
• [x] mimick the protocol for ByteFlies
• [x] setup a ByteFlies DAG for potential use with Apache Airflow
• [ ] adapt the 4 stages (pull meta, pull data, preprocess data, upload data)

Please comment/add any steps I am missing here. Will potentially set up a project with milestones for this issue.

Docs on https://ifu.byteflies.net/docs/dev/api-reference.html

[davidverweij]

@jawrainey @ColinBD I thought to record any design challenges and considerations here for us to discuss. For example, the ByteFlies API requires a username and password (to retrieve a token to) get access. This is at the moment my personal log in details. When we run this on the server - this feels less than ideal. Shall I inquire about a possible service account? Or do we already have one?

edit Also, a lot of patientIDs are missing in the ByteFlies portal. We could:

• ask clinicians to update these appropriately
• update these programmatically using the ByteFlies API

The patient IDs are also variously formatted - which requires a transformation and validity check in code.