kalekundert / stepwise

Modular, command-line scientific protocols
GNU General Public License v3.0
3 stars 0 forks source link

Cache information in pickles to speed up pipelines. #19

Closed kalekundert closed 4 years ago

kalekundert commented 4 years ago

I think that scanning the filesystem for protocols may be relatively slow. It becomes most noticeable in long pipelines, where each command needs to repeat this scan. The simplest way to speed this up is to have the first process cache the paths to all the protocols and to make this available to other processes via the pickle.

My first step would be to run the profiler, because I don't actually know if searching the filesystem takes an significant amount of time (although I suspect it does). Whatever the profiler finds, I might be able to cache in this manner.