denshoproject / ddr-local

Web UI used for interacting with DDR collections and entities on a local machine.
Other
3 stars 0 forks source link

Config for disabling slow processes for large collections #298

Open gjost opened 2 years ago

gjost commented 2 years ago

ddr-local runs git status under the hood in lots of places, mostly to get collection information. This is ok in most collections but causes problems in large collections like ddr-densho-1000 even with local/non-NFS files.

Running git status from the command-line can take several minutes, much longer than a single request-response cycle. This time is mostly spent in the Refresh index step. This doesn't happen every time tho -- once the index is refreshed, git status is pretty snappy.

Speedups might be possible depending on how many other things depend on git status. One idea would be to put the git status call in a timeout; if the call takes to long it could be run in the background by Celery and the results cached for a time. Or maybe it should always be run in the background? Or it could be run on a schedule, or only after certain read/write operations. Maybe a config setting could instruct ddr-local to always run it in the background.