search

CSB5 / lofreq

LoFreq Star: Sensitive variant calling from sequencing data
http://csb5.github.io/lofreq/
Other
100 stars 30 forks source link

LOFreq Improvements for HPC - buffers #91

Open lflis opened 4 years ago

lflis commented 4 years ago

Greetings from ACC Cyfronet HPC Software Team

Recently due to COVID outbreak number of genomic jobs on our clusters has significantly raised. Unfortunately most of the codes are not dealing well with distributed parallel filesystems like Lustre. This leads to resource underutilization long waiting for results and general frustration

Here is the first set of changes proposed to improve I/O for LOFREQ:

using big buffer for reading and writing:

In our case job cpu efficiency has rised from 60% to 100% by eliminating disk waits caused by lots of small i/o operations.

Please review proposed changes and if accepted please credit them to Cyfronet HPC Software Team and Maciej Czuchry m.czuchry (at) cyfronet.pl

hpc-buffers.patch.gz

andreas-wilm commented 4 years ago

Thanks for sharing @lflis. Interesting results.

I'm a bit puzzled by the effects you see. Your buffer length changes almost exclusively affect - assuming a normal workflow - the buffer that's used to write a vcf entry. That buffer size shouldn't affect the number of syscalls at all. Increasing it will only allow to hold larger entries, but I don't see how this would speed up anything. The fgets to fread change should, if at all, only affect vcf reading. Are your vcf files very large or are these covid sequences?

Many thanks for explaining

lflis commented 4 years ago

@andreas-wilm the case we were debugging today was some QBRC PIPELINE which on some stage calls lofreq for 26G vcf file.

By looking at ltrace and strace we observed that fgets was called with 4k block, which was translated to buffered 8k read(). In that part of workflow processes were waiting for I/O operations to complete for 40% of it's time.

Raising LINE_BUF_SIZE from 1<<12 to 1<<20 resulted in fgets being called with 1MB argument but this still translated to 8k read() calls. We have then replaced fgets with fread to avoid this buffering effect. As a result strace finaly shown expected 1MB read() calls.

In this case we effecitvely replaced every 128 read(8k) calls with one read(1M) call what means huge gain when dealing files located on lustre.

I hope that explains a bit.

andreas-wilm commented 4 years ago

Thanks for explaining. Allow me to ask two more questions:

  1. Doesn't your fgets to fread change actually change functionality and instead of reading one line at a time, it reads past the newline character, thus skipping entries? Do you get the same results before and after introducing your change?
  2. A vcf file size of 26GB sounds excessive, possibly because this file isn't bgzipped, which would be best practice. May I ask where this comes from and what it's used for? An obvious optimization would be to bgzip and index the file, depending on your workflow

Best

lflis commented 4 years ago

good points, please let me verify it

lflis commented 4 years ago

My appoligies for causing confusion. You are absolutely right with fgets - which is reading line by line. We must have failed testing somehere. Please ignore the patch. We'll get back here as soon as we implement proper buffered reads with a better testing.

lflis commented 4 years ago

@andreas-wilm from your experience - what are the most commonly used vcf and bgzipped vcf file sizes?.

andreas-wilm commented 4 years ago

That very much depends on the organism you are working with. From memory I'd say that dbsnp was 7GB.

On Thu, 16 Apr 2020 at 00:06, lflis notifications@github.com wrote:

@andreas-wilm https://github.com/andreas-wilm from your experience - what are the most commonly used vcf and bgzipped vcf file sizes?.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/CSB5/lofreq/issues/91#issuecomment-614130278, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAILSCKGPH2Q6IYNRHR2BKDRMXLQHANCNFSM4MITSGDA .

-- Andreas Wilm andreas.wilm@gmail.com | 0x7C68FBCC