Many of the data processing steps can be executed in parallel, however some steps particularly in the engine data portion of the pipeline are currently unnecessarily serial.
In particular, the cache-builder step creates one output at a time although those files are independent, reorganizing the snakefile and possibly revising corresponding engine program would speed this bottleneck.
Other steps in this part of the process may also lend themselves to similar reorganization (needs investigation).
Many of the data processing steps can be executed in parallel, however some steps particularly in the engine data portion of the pipeline are currently unnecessarily serial.
In particular, the cache-builder step creates one output at a time although those files are independent, reorganizing the snakefile and possibly revising corresponding engine program would speed this bottleneck.
Other steps in this part of the process may also lend themselves to similar reorganization (needs investigation).