search

coderxio / sagerx

Open drug data pipelines curated by pharmacists.
https://coderx.io/sagerx
Other
47 stars 12 forks source link

Full Dailymed DAG runs into error #238

Open jrlegrand opened 9 months ago

jrlegrand commented 9 months ago

Problem Statement

When trying to run DailyMed Full DAG (50,000 SPLs), I run into the error shown below.

This is from the build in branch jrlegrand/dailymed

Criteria for Success

Able to run DailyMed DAG to completion and have all 50,000 SPLs in the database.

Additional Information

I think this page might have info. Might be a DailyMed buffer setting. https://www.postgresql.org/docs/9.1/runtime-config-resource.html

This is the convo I had with Kent about this. https://coderx.slack.com/archives/C05S27E52N8/p1702217962520539?thread_ts=1701463757.571009&cid=C05S27E52N8

We can increase the buffer but I think there is some other issue here, from what I can tell the buffer is just a short term memory to improve speed but things if done correctly shouldn't be dependent on it

I'm honestly not sure if the error is about bad XML or truly about the buffer being full.

image (1)

image

jrlegrand commented 6 months ago

I wonder if the move to BigQuery will resolve this limitation...