artic-network / fieldbioinformatics

The ARTIC field bioinformatics pipeline
MIT License
110 stars 68 forks source link

Add multiprocessing capability to guppyplex #29

Open MarkusHaak opened 4 years ago

MarkusHaak commented 4 years ago

Adds multiprocessing to guppyplex functionality to accelerate the analysis of large datasets. The subprocesses of the worker pool each process one fastq file at a time and write their results in temporary fastq files named .tmp_[worker_id] to reduce memory load. These are then merged with sensitivity to duplicates, as in the original implementation. The default number of processes is set to 4.