how to use multiple threads with using this library for better speed of reading fastq files ?

attractivechaos / klib

A standalone and lightweight C library

http://attractivechaos.github.io/klib/

MIT License

4.18k stars 556 forks source link

how to use multiple threads with using this library for better speed of reading fastq files ? #137

Open lidd77 opened 4 years ago

lidd77 commented 4 years ago

Hi, In order to improve the speed of reading fastq files and doing some quality control statistics , wanna use multiple threads to complete some statistics of every four fastq lines from the FASTQ files . so could you recommend some methods/codes to realize that goal ? it's very slow to read fastq files in the way line by line .

Best!

ahcm commented 4 years ago

Read in big chunks in main thread. Process in memory in parallel.

Probably not worth optimization: let main thread distribute the chunks to the fitting numa nodes.