search

PennTURBO / medication-knowledgegraph-pipeline

Pipeline that maps strings describing medication orders to RxNorm terms, and generates an RDF knowledgegraph that links those strings to multiple ontologies including DrOn, ChEBI, ATC and NDF-RT.
4 stars 1 forks source link

CURL link between newest R med mapping scripts and RxNav Container lost after ~ 10k approximate match searches #21

Closed turbomam closed 3 years ago

turbomam commented 4 years ago

The R scripts for the training and classifying stages of the current medication mapping workflow submit medication strings to RxNav's approximate match search via CURL/REST. As good citizens, we submit these searches to a local RxNav-in-a-box Docker container.

After roughly 10k approximate match searches, the R script becomes unresponsive. Even at that point, approximate match searches do receive a response, if submitted to the same RxNav endpoint but via the curl command from the client's Bash prompt.

This limitation may be decreasing the training accuracy and will certainly make it inconvenient to map all of the medication strings from Penn's PDS clinical data warehouse. There's ~ 900k total and 250k that have an order/encounter link to an EMPI-identifiable person.

After completing a RDF instantiations of the current medication mapping CSV results, I'll get back to whittling the search code down to a minimal example and adding better logging.

turbomam commented 3 years ago

applied batching of request with a brief sleep