with the recent updates to bib pipeline (switched off reading references), we can clock 400 recs/s -- with bursts of 600 recs/s
so it definitely improved the speed; but the merge metadata is slowing things down; is able to process 150 recs/s on average.
we could try to give more processes to the merge-metadata queue, but not before the code is examined and checked for obvious optimizations. and hopefully profiled
with the recent updates to bib pipeline (switched off reading references), we can clock 400 recs/s -- with bursts of 600 recs/s
so it definitely improved the speed; but the merge metadata is slowing things down; is able to process 150 recs/s on average.
we could try to give more processes to the merge-metadata queue, but not before the code is examined and checked for obvious optimizations. and hopefully profiled