scaife-viewer / backend

Packages and utilities to build Scaife Viewer backends using ATLAS / CTS resolvers
3 stars 2 forks source link

[atlas] Improve bulk ingestion performance #39

Closed jacobwegner closed 3 years ago

jacobwegner commented 3 years ago

Follows up scaife-viewer/sv-mini-atlas#10 (Speed up ingestion using bulk insert / updates).

For corpora with a large number of work-level nodes, skipping calculation of Node.numchild can speed up ingestion by roughly 180%.

Since both the tree path and URN capture heirarchy, we can work around the standard django-treebeard methods for the "read" queries.