search

phac-nml / rebar

REcombination BARcode detector.
https://phac-nml.github.io/rebar/
Apache License 2.0
12 stars 1 forks source link

input: stream alignment and show progress bar #13

Open ktmeaton opened 9 months ago

ktmeaton commented 9 months ago

When working with large datasets (VirusSeq) it would be nice to monitor progress. We could also use multithreading for it (I think), if we don't mind the output being in a different order from the input.

ktmeaton commented 9 months ago

Since reading a file is inherently a linear operation, multi-threading won't be appropriate there. But potentially we can stream the file reading directly into the recombination search, with par_bridge: https://morestina.net/blog/1432/parallel-stream-processing-with-rayon#Stream_processing