Not fully thought out, but I like the idea of incorporating the idea of "recipes" into the ETL process. A recipe would be a structured package of information that describes the process of transforming data from a source format to MEDS.
People would be able to share recipes for different datasets. e.g. there would be a recipe for MIMIC-IV v2.2 -> MEDS, and perhaps track these recipes in a central repository.
Refactoring to incorporate these recipes would help to separate the custom data descriptions from the code that implements the transform (which appears to be duplicated across scripts at the moment).
Not fully thought out, but I like the idea of incorporating the idea of "recipes" into the ETL process. A recipe would be a structured package of information that describes the process of transforming data from a source format to MEDS.
People would be able to share recipes for different datasets. e.g. there would be a recipe for MIMIC-IV v2.2 -> MEDS, and perhaps track these recipes in a central repository.
Refactoring to incorporate these recipes would help to separate the custom data descriptions from the code that implements the transform (which appears to be duplicated across scripts at the moment).