Parallel Processing Option

YeoLab / merge_peaks

Pipeline for using IDR to produce a set of peaks given two replicate eCLIP peaks

9 stars 7 forks source link

Parallel Processing Option #13

Closed BuddahKat closed 11 months ago

BuddahKat commented 3 years ago

Is there a multi-threading option for the 'merge_peaks' pipeline? Just wondering if it's possible to speed it up a bit, it took 24 hours to complete and my BAM files aren't that big. I think the bottleneck is the IDR component which seems to call 'clipper' repeatedly, I know clipper is able to be run in parallel so is the anyway to pass the '--processors' flag to clipper without having to modify a bunch of files?

byee4 commented 3 years ago

I think clipper tries to use as many threads as possible without the --processors flag, so the way to speed this up would be to have more processors available. For example, I've noticed a substantial increase when running Clipper on a 16 ppn and a 32 ppn node.