search

wolfgangw / backports

Deep inspection of digital cinema packages
27 stars 12 forks source link

Speed up search for AssetMap candidates and initial processing #94

Closed wolfgangw closed 8 months ago

wolfgangw commented 9 months ago

Startup over a large dirtree is unacceptably slow. Find a better way.

wolfgangw commented 8 months ago

9769fe4b0a670a02ad988e8a87badef61e53e9eb is a stab at that. By a) using fdfind and b) going directly for AM candidates I'm seeing a significant speedup. Needs more testing over large trees.

This changes used and displayed asset paths from relative to full. Not quite sure yet how I like that. See related package() and clean that up.

wolfgangw commented 8 months ago

Run over a 40 TB tree with ~1000 AMs/packages survived. Initial AM search is way faster with fdfind and the explicit search.

Looking ok so far. Needs more testing for completeness of AM search result.

wolfgangw commented 8 months ago

Add an interim timer after AM search

wolfgangw commented 8 months ago

Initial AM search over the 40 TB tree ~4 min. Look into stream processing of fdfind's progress to get stuff going faster.

See e.g. https://mensfeld.pl/2016/12/ruby-stream-processing-of-shell-command-results/

wolfgangw commented 8 months ago

Out of curiosity and as yet another tidbit: report number of AM candidates vs. number of actual AMs.