Medical-Event-Data-Standard / meds_etl

A collection of ETLs from common data formats to Medical Event Data Standard
Apache License 2.0
16 stars 3 forks source link

Add optional native backend to meds_etl #16

Closed EthanSteinberg closed 2 months ago

EthanSteinberg commented 2 months ago

The current ETLs within meds_etl work, but they are quite slow.

This adds an optimized implementation, written in C++ as an optional path.

The optimized code is within https://github.com/Medical-Event-Data-Standard/meds_etl_cpp.

From my simple tests, the newer code is roughly 100x faster than the old polars implementation.