RIAB: Performance RiAB & cost

RADar-AZDelta / Rabbit-in-a-Blender

An ETL pipeline to transform your EMP data to OMOP.

https://radar-azdelta.github.io/Rabbit-in-a-Blender/

GNU General Public License v3.0

11 stars 3 forks source link

RIAB: Performance RiAB & cost #31

Closed kdenturc closed 7 months ago

kdenturc commented 8 months ago

@pjlammertyn running RiAB on measurement table alone has taken runtime approximate 15 to 20 minutes and at the end the cost went up to 10EUR, so today I've spent over 100EUR. How can we optimize?

If I took the labo_usagi_csv file and diminished the rows from 6500 till 70, then runtime went to 3 minutes and cost was 0.91EUR which is funner to work with.

Would it be an idea for performance and cost that we delete unchecked and flagged mappings? how can we do so?

tx! please follow @sieldep

sieldep commented 8 months ago

De usagi files voor mzg diagnoses zijn bijvoorbeeld ook heel groot. Eventueel ook filteren op wanneer sourceFrequency 0 is? Dat deze niet in de usagi blijven staan?