search

wolfgangw / backports

Deep inspection of digital cinema packages
27 stars 12 forks source link

Speed up search for AssetMap candidates and initial processing #94

Closed wolfgangw closed 10 months ago

wolfgangw commented 11 months ago

Startup over a large dirtree is unacceptably slow. Find a better way.

wolfgangw commented 11 months ago

9769fe4b0a670a02ad988e8a87badef61e53e9eb is a stab at that. By a) using fdfind and b) going directly for AM candidates I'm seeing a significant speedup. Needs more testing over large trees.

This changes used and displayed asset paths from relative to full. Not quite sure yet how I like that. See related package() and clean that up.

wolfgangw commented 11 months ago

Run over a 40 TB tree with ~1000 AMs/packages survived. Initial AM search is way faster with fdfind and the explicit search.

Looking ok so far. Needs more testing for completeness of AM search result.

wolfgangw commented 11 months ago

Add an interim timer after AM search

wolfgangw commented 10 months ago

Initial AM search over the 40 TB tree ~4 min. Look into stream processing of fdfind's progress to get stuff going faster.

See e.g. https://mensfeld.pl/2016/12/ruby-stream-processing-of-shell-command-results/

wolfgangw commented 10 months ago

Out of curiosity and as yet another tidbit: report number of AM candidates vs. number of actual AMs.