Cleaned up DAG based on new airflow standard.
Changed RXCUIs to look at ingredient level reducing the number of API calls from 70k to 18k.
Added concurrency logic to run multiple API calls at once.
Fix table column naming
Reduces runtime from 5 hours to 6 minutes for RxClass
Reduces runtime from 5 hours to 20 min for RxNorm Historical
Tests
What testing did you do?
Ran DAG to completion. Checks PG database for correct count and got 24227 rows: SELECT COUNT(*) FROM datasource.rxclass_atc_to_product
Resolves #256, #250, #218, #265
Explanation
Cleaned up DAG based on new airflow standard. Changed RXCUIs to look at ingredient level reducing the number of API calls from 70k to 18k. Added concurrency logic to run multiple API calls at once. Fix table column naming
Reduces runtime from 5 hours to 6 minutes for RxClass
Reduces runtime from 5 hours to 20 min for RxNorm Historical
Tests
SELECT COUNT(*) FROM datasource.rxclass_atc_to_producttesting logs
``` ```