search

fsdonks / proc

MARATHON post proccessor
0 stars 0 forks source link

Use of I/O in proc.core/only-srcs performance hit #20

Closed fs-tom closed 6 years ago

fs-tom commented 6 years ago

There are repeated calls to proc.core/srcs+, defined in the bindings for filterf# etc. inside proc.core/only-srcs. This likely causes significant performance implications, since we touch the disk (in srcs+) via spork.util.table/tabdelimited->records, and recompute the same set for every record.....

Quick fix: lift the computed sets out into additional binds in the lexical scope, substitute bound vars for original computations. This should cache the set, only touch disk once, and generally be much faster....

fs-tom commented 6 years ago

patched. we compute the sets only once, avoiding repeated i/o.