Export dag

[leemlb06pmi]

Resolves #262, #266

Explanation

[What did you change?] Added two new DAGs to first run dependencies and then rebuild the Mart Models then second to create and export .csv files from those mart tables. We've also added a few small changes to optimize the docker-compose file

For Future Reference: Once we implement a freshness parameter, we can update the transform task so that the freshness is checked before certain tables are re-built.

Rationale

[Why did you make the changes mentioned above? What alternatives did you consider?]

Tests

1. What testing did you do?
2. Attach testing logs inside a summary block:

testing logs

``` ```

[jrlegrand]

I ran rxnorm to completion (as well as all 3 FDA DAGs) and then ran build_marts and it started running rxnorm again, but not the FDA DAGs.

[jrlegrand]

Got this error in the rxnorm_historical DAG. I've never seen it before. Going to try to re-run and see if I get it again. Might not be related at all to this PR.

[2024-04-17, 18:07:22 UTC] {local_task_job.py:273} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2024-04-17, 18:07:22 UTC] {process_utils.py:133} INFO - Sending Signals.SIGTERM to group 9670. PIDs of all processes in the group: [9670]
[2024-04-17, 18:07:22 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 9670
[2024-04-17, 18:07:22 UTC] {taskinstance.py:1479} ERROR - Received SIGTERM. Terminating subprocesses.
[2024-04-17, 18:08:22 UTC] {process_utils.py:147} WARNING - process psutil.Process(pid=9670, name='airflow task ru', status='sleeping', started='18:06:46') did not respond to SIGTERM. Trying SIGKILL
[2024-04-17, 18:08:22 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGKILL to group 9670
[2024-04-17, 18:08:22 UTC] {process_utils.py:79} INFO - Process psutil.Process(pid=9670, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='18:06:46') (9670) terminated with exit code Negsignal.SIGKILL
[2024-04-17, 18:08:22 UTC] {standard_task_runner.py:170} ERROR - Job 46 was killed before it finished (likely due to running out of memory)

[leemlb06pmi]

I ran rxnorm to completion (as well as all 3 FDA DAGs) and then ran build_marts and it started running rxnorm again, but not the FDA DAGs.

This will probably be because if the DAGs were not turned on before now (RxNorm) then the first run logs a run date = start date, so I think for RxNorm that is sometime in March? Either way, I can add some logging to display what the last run date recorded is for future reference - in case something doesn't get triggered that we think should be

[leemlb06pmi]

Got this error in the rxnorm_historical DAG. I've never seen it before. Going to try to re-run and see if I get it again. Might not be related at all to this PR.

[2024-04-17, 18:07:22 UTC] {local_task_job.py:273} WARNING - State of this instance has been externally set to failed. Terminating instance.
[2024-04-17, 18:07:22 UTC] {process_utils.py:133} INFO - Sending Signals.SIGTERM to group 9670. PIDs of all processes in the group: [9670]
[2024-04-17, 18:07:22 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 9670
[2024-04-17, 18:07:22 UTC] {taskinstance.py:1479} ERROR - Received SIGTERM. Terminating subprocesses.
[2024-04-17, 18:08:22 UTC] {process_utils.py:147} WARNING - process psutil.Process(pid=9670, name='airflow task ru', status='sleeping', started='18:06:46') did not respond to SIGTERM. Trying SIGKILL
[2024-04-17, 18:08:22 UTC] {process_utils.py:84} INFO - Sending the signal Signals.SIGKILL to group 9670
[2024-04-17, 18:08:22 UTC] {process_utils.py:79} INFO - Process psutil.Process(pid=9670, name='airflow task ru', status='terminated', exitcode=<Negsignal.SIGKILL: -9>, started='18:06:46') (9670) terminated with exit code Negsignal.SIGKILL
[2024-04-17, 18:08:22 UTC] {standard_task_runner.py:170} ERROR - Job 46 was killed before it finished (likely due to running out of memory)

This may very well be a memory issue - was this a local run? I imagine in the past you've let rxnorm_historical by itself, but in this case when it is triggered by the marts dag, there would be another dag running at the same time which could potentially cause a memory overflow. I have my local instance of Docker configured to use up to 8gb of memory - could you check what yours is set to?

[jrlegrand]

I ran rxnorm to completion (as well as all 3 FDA DAGs) and then ran build_marts and it started running rxnorm again, but not the FDA DAGs.

This will probably be because if the DAGs were not turned on before now (RxNorm) then the first run logs a run date = start date, so I think for RxNorm that is sometime in March? Either way, I can add some logging to display what the last run date recorded is for future reference - in case something doesn't get triggered that we think should be

I'm not too concerned with it right now. I think I will leave this on for a week or two and put in an issue if it becomes a problem. If you want to add some logging, we might be able to track down what's happening though.