search

MikeAxtell / CleaveLand4

CleaveLand4: Analysis of degradome data to find sliced miRNA and siRNA targets
GNU General Public License v3.0
19 stars 13 forks source link

Parallel computing #4

Open netbofia opened 8 years ago

netbofia commented 8 years ago

Hello this is a great tool but it takes a long time to run. Is it possible to parallelize it. For example running it n time by dividing the query miRNAs and feeding them to the multiple running instances.

MikeAxtell commented 8 years ago

Thanks. Yes, it's slow. You can do a poor man's parallel processing by splitting miRNA query files as you say and submitted to separate jobs on a cluster.

Cheers, Mike

On Sat, May 28, 2016 at 3:34 PM, Bruno Costa notifications@github.com wrote:

Hello this is a great tool but it takes a long time to run. Is it possible to parallelize it. For example running it n time by dividing the query miRNAs and feeding them to the multiple running instances.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MikeAxtell/CleaveLand4/issues/4, or mute the thread https://github.com/notifications/unsubscribe/AGiXiU_sVb93z1aPMlnpVQjG-VEeX7frks5qGJirgaJpZM4IpJMH .

Michael J. Axtell, Ph.D. Associate Professor Dept. of Biology Penn State University http://sites.psu.edu/axtell Twitter: @MikeAxtell

netbofia commented 8 years ago

Thanks for the feedback. I've forked the project and created a wrapper in bash for Cleaveland4 It allows running in mode 1 & 2. (Which I think are the only ones that make sense parallelizing) Grabs x Sequences from -u makes a new directory and runs. Afterward all outputs are merged together.

Most code is commented should be easy to understand I hope.

Is this something you would be interested in merging with your work. Check my fork: https://github.com/netbofia/CleaveLand4

Bruno

MikeAxtell commented 8 years ago

Cool, I'll check it out next week after I get out from under a grant deadline and a trip.

Cheers, Mike

On Tue, May 31, 2016 at 3:22 PM, Bruno Costa notifications@github.com wrote:

Thanks for the feedback. I've forked the project and created a wrapper in bash for Cleaveland4 It allows running in mode 1 & 2. (Which I think are the only ones that make sense parallelizing) Grabs x Sequences from -u makes a new directory and runs. Afterward all outputs are merged together.

Most code is commented should be easy to understand I hope.

Is this something you would be interested in merging with your work. Check my fork: https://github.com/netbofia/CleaveLand4

Bruno

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MikeAxtell/CleaveLand4/issues/4#issuecomment-222792847, or mute the thread https://github.com/notifications/unsubscribe/AGiXiaEr-iuG5gO7ak_ei2NzFjPffBXQks5qHIqBgaJpZM4IpJMH .

Michael J. Axtell, Ph.D. Associate Professor Dept. of Biology Penn State University http://sites.psu.edu/axtell Twitter: @MikeAxtell

wangyugui commented 7 years ago

@netbofia the fork of netbofia is version 4.3 only.

@MikeAxtell I tried to split miRNA query files with the command '/usr/bin/parallel', but it seem that we need something to do about temp file inside the process.

netbofia commented 7 years ago

Sorry only saw this today. I'll test it and give you feed back in the following days.

netbofia commented 5 years ago

@MikeAxtell @wangyugui I'm sorry for this big delay. @wangyugui The fork can work with any version of the input scheme is the same. Just change the cleaveLand var in line 169 to point to the cleaveland file of choice.

Let me know about any issues. I tested this version and it seams to hold up but strange issues might arise. Send me an email if necessary at: brunocosta@itqb.unl.pt