search

OpenGene / fastp

An ultra-fast all-in-one FASTQ preprocessor (QC/adapters/trimming/filtering/splitting/merging...)
MIT License
1.95k stars 334 forks source link

Implement vectorisation to make this lib blazingly fast #585

Open chinwobble opened 4 days ago

chinwobble commented 4 days ago

Vectorisation using SIMD should provide big performance improvements on large datasets. There are a few candidates for vectorisation such as in the merge function. https://github.com/OpenGene/fastp/blob/master/src/stats.cpp#L881-L939

I'm happy to have a go at implementing this if you can provide a sufficient fastq file for me to test with

References: https://chryswoods.com/vector_c++/portable.html https://github.com/google/highway/

sfchen commented 4 days ago

Thanks, but currently the merge function is not the bottleneck of performance.

sfchen commented 4 days ago

And I completely agree with you that SSE/AVX can make this tool even faster. You can take a look at fastplong, which is more time consuming and is being intensively developed.

teepean commented 4 days ago

@chinwobble AdapterRemoval got a nice speed boost when SIMD instructions were introduced. It is a similar program to fastp so you might get the idea where the speed boost is needed.

https://github.com/MikkelSchubert/adapterremoval

chinwobble commented 2 days ago

And I completely agree with you that SSE/AVX can make this tool even faster. You can take a look at fastplong, which is more time consuming and is being intensively developed.

Thanks I will have a go at implementing this! Can you provide some large test files for me to work with?