aseyboldt / fastq-rs

MIT License
47 stars 12 forks source link
fastq

Build Status

A fast parser for fastq.

This library can process fastq files at about the speed of the coreutils wc -l (about 2GB/s on my laptop, seqan runs at about 150MB/s). It also makes it easy to distribute the processing of fastq records to many cores, without losing much of the performance.

See the documentation for details and examples.

Benchmarks

We compare this library with the fastq parser in rust-bio, the C++ library seqan 2.2.0, with kseq.h and with wc -l.

We test 4 scenarios:

All measurements are taken with a 2GB test file (TODO describe!) on a Haskwell i7-4510U @ 2GH. Each program is executed three times (clearing the os page cache where appropriate) and the best time is used. Libraries without native support for a compression algorithm get the input via a pipe from zcat or lz4 -d. The C and C++ programs are compiled with gcc 6.2.1 with the fags -O3 -march=native. All programs can be found in the examples directory of this repository.

ramdisk lz4 lz4 + thread gzip gzip + thread
wc -l 2.3GB/s 1.2GB/s NA 300MB/s NA
fastq 1.9GB/s 1.9GB/s 1.6GB/s 650MB/s 620MB/s
rust-bio 730MB/s NA 250MB/s NA NA
seqan 150MB/s NA NA NA NA
kseq.h 980MB/s 680MB/s NA NA NA

Some notes from checking perf record: