search

google / fhir-data-pipes

A collection of tools for extracting FHIR resources and analytics services on top of that data.
https://google.github.io/fhir-data-pipes/
Apache License 2.0
141 stars 80 forks source link

Compare Spark+Parquet with PostgreSQL/Alloy DB view based approaches #968

Open bashir2 opened 4 months ago

bashir2 commented 4 months ago

Now that our support for SQL-on-FHIR-v2 ViewDefinition is complete (#821 and #916) we should do some large scale comparisons of the Spark+Parquet based approach with relational DB based ones using materialized views. We can start with PostgreSQL and establish some guidelines on the scale of data at which using a single node PG DB starts to make less sense (compared to a multi node Spark+Parquet based approach). Then we should repeat the same experiment with AlloyDB to see the impact of columnar storage (while still single node).

We should do several experiments using multiple realistic workloads (e.g., calculating program or data quality metrics involving joins of multiple resource tables). But we also recognize that these comparisons will always be subjective to some extent, because of the choice of workloads.

jakubadamek commented 4 months ago

I might be interested as a sequel to #967

bashir2 commented 4 months ago

I might be interested as a sequel to #967

That would be great; please feel free to assign this to yourself once you start working on it.