Open ktmeaton opened 9 months ago
Since reading a file is inherently a linear operation, multi-threading won't be appropriate there. But potentially we can stream the file reading directly into the recombination search, with par_bridge
:
https://morestina.net/blog/1432/parallel-stream-processing-with-rayon#Stream_processing
When working with large datasets (VirusSeq) it would be nice to monitor progress. We could also use multithreading for it (I think), if we don't mind the output being in a different order from the input.