Closed victorkingi closed 3 years ago
We were given a spec from the client for our data ingestion and it used a RESTful OpenAPI design. This makes sense for EHRs that are created in-time. For historical data, we are better off using one of these two options. For realtime analytics, we could use a Kinesis Data Firehose
Hi, from our perspective this is the first time we've encountered any of these requirements. We were given a spec that defined a RESTful API only. I believed these would be enhancement proposals, unless they are hard requirements. If so, I would recommend that we split our teams to work on this integration separately from our core sprint.
By the looks of it, we can add SFTP support in a future version by simply adding a new staging bucket. We would need to ensure our ETL process produces something consistent when added to the lake.
In terms of message broker, essentially you are either talking about having FHIR messages sent to a server (which can just be the API we originally provided) or we provide a streaming server (like Kafka) but this will in fact be Kinesis.
Are they your requirements or our requirements? Because we can help you work on the same cloud environment if you want to build this stuff now. I'm not aware that the client requested this of the data lake team.
Let me consult the team and get back to you
How would having a streaming server look like?
Describe the issue As part of the data simulator To-Do, we were to simulate at least 4 modes of data transport including scheduled SFTP, message broker technology, HTTPS and data federation. The HTTPS is sorted since we sent POST requests successfully, we now need a public key and login details for SFTP. We are still researching on data federation and message broker so the requirements will be added by updating this issue.
Desktop: