search

mckennalab / FlashFry

FlashFry: The rapid CRISPR target site characterization tool
Other
63 stars 10 forks source link

Run with multiple threads/cores? #36

Closed nkuperwasser closed 3 months ago

nkuperwasser commented 1 year ago

Thank you very much for the creation of this program, works great! I am starting to assemble some fairly large regions to run the discover module, and I was wondering if/how it would be possible to run with multiple threads/cores?

Thanks!!

Sincerely,

NK

acrinklaw commented 1 year ago

I've run some fairly absurd things through FlashFry (millions of FASTA sequences against entire genomes, etc) and found good results by just splitting up the input and parallelizing with either Snakemake or GNU parallel.

nkuperwasser commented 1 year ago

Okay, thank you for the suggestion! Will look into snakemake and start working with it. I apologize in advance if this question sounds a bit naive (it's trial by fire for me in terms of python,r,unix,etc etc), but if I parallelize, is there a general consensus for the memory flags for the java instances and the size of the corresponding fasta files?

Sincerely, NK

acrinklaw commented 1 year ago

This is something that depends on your compute resources, input size, database size, etc. FlashFry already uses concurrency, I have just found that at some large input sizes for large databases with many off targets it can hang on trying to write out all the data. Maybe the author has done some benchmarking and can weigh in better on requirements

nkuperwasser commented 1 year ago

Okay..thank you! I've been trying to run variations on the machine (via ssh) and twice in a row, I get a "broken pipe" and logged out...I will keep on trying to reduce the fasta sizes....(trial by fire :) )

acrinklaw commented 1 year ago

Okay..thank you! I've been trying to run variations on the machine (via ssh) and twice in a row, I get a "broken pipe" and logged out...I will keep on trying to reduce the fasta sizes....(trial by fire :) )

Try giving tmux a try...it will preserve your session in case your SSH pipe breaks. tmux tutorial

nkuperwasser commented 3 months ago

Haven't had a chance to properly use tmux...got a newer silicon chip mac and have had some issues trying to get things back into order with various conda/python and other associated issues.

Sounds promising, and will start to do that soon.