Improve build speeds on large projects?

VUIIS / dax

Distributed Automation for XNAT

MIT License

25 stars 24 forks source link

Improve build speeds on large projects? #429

Open baxpr opened 1 year ago

baxpr commented 1 year ago

ABCD has ~17000 sessions and builds take about two months.

Each session in the current build is taking 8-9 minutes. Wondering if there's any way to consolidate REDCap or XNAT queries within a session or across sessions to speed things up.

baxpr commented 1 year ago

Or parallelize builds somehow? Bookkeeping and conflict management would be needed - lock field somewhere in an XNAT record, and how to avoid a race condition with the locking?