sul-dlss-deprecated / dor_indexing_app

An indexing API for Stanford's Digital Object Repository

https://sul-dlss-deprecated.github.io/dor_indexing_app/

Apache License 2.0

0 stars 2 forks source link

Spruce up, and dial back, rolling indexer #1046

Closed mjgiarlo closed 1 year ago

mjgiarlo commented 1 year ago

Why was this change made?

This commit makes the following changes to the rolling indexer:

Limit parallelization within the indexer to 2 processes so we don't take down services under load
Replace magic numbers in rolling indexer with values from settings for easier per-env
Enable logging within rolling indexer
Write pidfile to a fixed location for easier management of the rolling_index process
Log the total runtime of each iteration of the rolling indexer

How was this change tested?

[x] CI
[x] Deployed to stage, restarted indexer, watched log, saw expected output:

2023-10-18 12:11:16 -0700       Indexed 500 documents in 34.766
2023-10-18 12:11:56 -0700       Indexed 500 documents in 34.877
2023-10-18 12:12:36 -0700       Indexed 500 documents in 35.325
2023-10-18 12:13:17 -0700       Indexed 500 documents in 35.562
2023-10-18 12:13:59 -0700       Indexed 500 documents in 35.156

So this is running a bit more quickly than perhaps we expected, and maybe the 2x improvement is sufficient?