Support for storing parquet files in AWS S3 is not available

google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.

https://google.github.io/fhir-data-pipes/

Apache License 2.0

140 stars 80 forks source link

Support for storing parquet files in AWS S3 is not available #1011

Open Charantl opened 2 months ago

Charantl commented 2 months ago

bashir2 commented 1 month ago

I think Beam supports S3 natively; we should probably just add a dependency and maybe set some filesystem options to fix this. I will take a look at this soon.

Charantl commented 1 week ago

S3 support issue

@bashir2, We are still facing the same issue while applying the S3 location in dwhRootPrefix of docker/config/application.yaml file.

bashir2 commented 1 week ago

Thanks @Charantl for the note; when I fixed this last month I only tested it in the base pipeline and made sure S3 output locations work. This error is coming from the controller which I missed. I am going to fix it next week.