Increase processing speed by running queries in parallel

ontoinsights / deep_narrative_analysis

Ontologies and Python code to create a semantic and ML infrastructure to enable deeper exploration and understanding of narratives

Other

18 stars 2 forks source link

Increase processing speed by running queries in parallel #13

Open AndreaWesterinen opened 1 month ago

AndreaWesterinen commented 1 month ago

Currently, 9-10 mins per article ingested, partially processing all sentences and fully analyzing 10 sentences

AndreaWesterinen commented 1 month ago

Using summary sentences vs full, complex ones reduced processing time by 1.5x, but results in other difficulties (such as missing important info in sub-clauses, and mismatch of "trigger" text with actual text).

Still being investigated and still need to parallelize.