search

clingen-data-model / genegraph

Presents an RDF triplestore of gene information using GraphQL APIs
5 stars 0 forks source link

Stream ingest halts when db transaction fails in other thread #194

Open theferrit32 opened 3 years ago

theferrit32 commented 3 years ago

On a recent deploy of the genegraph-clinvar deployment, the ingest halted at topic offset 25192, around halfway through the topic clinvar-combined_20210302. From the log it appears immediately after the ingest of that clinvar message, an exception related to taoensso.nippy/thaw occurred on RXNORM data. After this nippy/RXNORM exception, the clinvar topic ingest does not resume.

log-trimmed.txt

theferrit32 commented 3 years ago

Genegraph migration was 2020-12-17T1429. I will update to a more recent one and retry.

larrybabb commented 3 years ago

@tnavatar @toneillbroad is there any chance it would help to only pull in the salient records in RXNORM for the purpose of both speeding up the ingest as well as reducing the potential for issues and burden on the repository? I have to believe that we only need < 5% of the total data we are loading. Let me know if this is something you'd like to do and if I need to discuss this with the website team and Erin.

theferrit32 commented 3 years ago

Updating the genegraph migration to 2021-04-22T1330 seems to have removed RXNORM errors from the pod log. I am not sure what has changed in the codebase related to RXNORM between those two dates but I'll keep using the more recent data version.

larrybabb commented 3 years ago

@theferrit32 should we close this for now and wait and see if it reemerges?

theferrit32 commented 3 years ago

@larrybabb it might be worth looking into why the exception on the RXNORM loading interfered with loading data from topics, because there's the potential a similar edge case could happen for something else in the future, but I'm fine with closing for now until it's observed again.

KelseaChang5 commented 2 years ago

December 7th 2021 Triage: Archived