Faster `convert_flat_to_meds`

Medical-Event-Data-Standard / meds_etl

A collection of ETLs from common data formats to Medical Event Data Standard

Apache License 2.0

16 stars 3 forks source link

Closed Miking98 closed 2 months ago

Miking98 commented 3 months ago

Parallelizes shard processing for Polars in convert_flat_to_meds
Adds support for using DuckDB instead of Polars in convert_flat_to_meds (reduces time to process EHRSHOT using 100 shards, 6 procs from 50 mins -> 3 mins)

scottfleming commented 3 months ago

Phenomenal work! Be sure to also add duckdb to the requirements/pyproject.toml

EthanSteinberg commented 2 months ago

I'm closing this as this has been made obsolete by Scott's PR