search

coderxio / sagerx

Open drug data pipelines curated by pharmacists.
https://coderx.io/sagerx
Other
41 stars 11 forks source link

all changes necessary to implement rxnorm indexes #281

Closed leemlb06pmi closed 2 months ago

leemlb06pmi commented 2 months ago

Resolves #270 Resolves #164

Explanation

We added all necessary SQL code and some additional airflow changes to implement the recommended indexes for rxnorm

Rationale

this would theoretically have upstream affects on all views/tables that reference these raw layer tables since they will now be more performant. We also introduced some additional changes to airflow that will increase concurrency amongst tasks, but which could introduce conflicts in other dags. recommend isolation testing in dev env for at least 1-2 full cycles of each dag.

Tests

  1. What testing did you do?
  2. Attach testing logs inside a summary block:
testing logs ``` ```
lprzychodzien commented 2 months ago

Should the create indexes be after the COPY command? Assuming the file is executed in order, it is CREATE TABLE -> load data with COPY -> CREATE INDEX.

lprzychodzien commented 2 months ago

Implemented the indexes locally and everything ran correctly and quickly.

image
leemlb06pmi commented 2 months ago

New commit added moving indexes to after the copy