apache / beam

Apache Beam is a unified programming model for Batch and Streaming data processing.
https://beam.apache.org/
Apache License 2.0
7.81k stars 4.23k forks source link

[Task]: Create Colab Notebook for `WriteToFhir` #29199

Open svetakvsundhar opened 11 months ago

svetakvsundhar commented 11 months ago

What needs to happen?

Currently, we are creating a PTransform for WriteToFhir, which allows Beam to write to a Fhir store in the Python SDK. Note that this capability exists in the Java SDK via the FhirIO, but not in the Python SDK.

Given the maintenance burden associated with adding new IOs, I propose we add PTransforms and connectors like this in the form of colab examples, rather than in the repo itself. This will decrease the risk of unmaintained code in the repo.

cc: @damccorm @johnjcasey @liferoad to get your thoughts on the above paragraph.

Issue Priority

Priority: 3 (nice-to-have improvement)

Issue Components

damccorm commented 11 months ago

Rather than implementing this from scratch, should we just add a cross language wrapper to the Java version? It seems like that might both accomplish the goal of reducing the maintenance/bug surface and giving users easy access to the functionality.

svetakvsundhar commented 11 months ago

+1, that'd be my first preference, but I'm not sure the effort required for that. @ahmedabu98 any thoughts here?

johnjcasey commented 11 months ago

I'd generally prefer xlang as well

ahmedabu98 commented 11 months ago

+1 to leaning on the xlang path.

Regarding the effort required, we would just have to create a SchemaTransform for the Java IO (can range from a day to a week depending on how familiar you are with the IO). There's a guide here that explains what is needed for that.

I'm currently working on a way to automatically generate Python wrappers so a well-documented SchemaTransform would be sufficient.