search

suhrig / arriba

Fast and accurate gene fusion detection from RNA-Seq data
Other
226 stars 49 forks source link

Multi-Threading #13

Closed akeeeshi closed 5 years ago

akeeeshi commented 5 years ago

Hi, just wanted to say how awesome this package is. I have found the results to be robust and comprehensive in terms of information provided (compared to any other package I've used).

I had quick question related to the multi-threading and whether there is any consideration around implementing it as flag option? Might speed up the processing time for those of using WGS/WES BAM files.

Thank you again!

suhrig commented 5 years ago

Hi, thanks for the positive feedback! Always good to hear that the tool is useful.

Regarding your request: I have considered support for multi-threading a number of times. So far I have always dismissed it for two reasons:

  1. Some of the steps would be hard to parallelize and others would hardly benefit from parallelization, because the overhead to dispatch the work to multiple threads would exceed the actual work that has to be done.

  2. Arriba is already pretty fast. Ideally the most time-consuming step (extracting reads from the BAM files) runs in parallel to alignment, so no time is lost. I understand that many users want to run Arriba separately from alignment, such that parallelized extraction of reads could reduce the runtime noticeably. But I have seen it process RNA-Seq samples with a billion reads in an hour (excluding alignment), which is still not much time.

I have never tried implementing multi-threading. So I don't actually know how much it would improve the speed, but my gut feeling tells me it is not worth pursuing. I'll reconsider, if your use case makes a compelling point that multi-threading would be of much benefit. What is your experience? Have you noticed particularly long runtimes with certain samples? This could be a bug that should be fixed, rather than using the steamroller approach "multi-threading". On a related note: The next release of Arriba comes with speed optimizations for two of the slower filters ("low_entropy" and "homologs") and faster parsing of GTF files.

BTW Arriba was not designed for DNA-Seq data (WGS/WES). It makes many assumptions which are only applicable to RNA-Seq data. If you want to find fusions in DNA-Seq data, you are probably better off using a structural variant caller. Here are a number of tools to choose from: https://bio.tools/?page=1&q=%27DNA%20structural%20variation%27&sort=score Once you have structural variant calls, you can feed them to Arriba as additional information (see option -d). But I would not recommend running Arriba on DNA-Seq data directly.

MattBashton commented 5 years ago

I would just like to add, arriba is really quite fast, so I don't think multi-threaded execution per analysis run is the right form of parallelism either. However, assuming you have a whole bunch of different RNA-Seq samples then a more appropriate form of parallelism would be to run arriba on a whole bunch of different samples in parallel at once using say GNU parallel, I have implemented such a system in a bash script which you might find useful.

suhrig commented 5 years ago

@akeeeshi: A new release is available. It comes with several speed improvements, most notably ~25% faster reading of BAM files.

I agree with @MattBashton that process-level parallelization using GNU parallel is the way to go. If you don't object, I will close this issue.

@MattBashton: Great script! So far I used to wrap Arriba in a call to xargs -P, but yours look much more elaborate. I will give it a try. ;-)

akeeeshi commented 5 years ago

I apologize for the delay in response. My Github notifications are doing something odd. After hearing everyone's input I can see how process-level parallelization is the way to go. Appreciate the help!