low memory usage

[anoronh4]

Despite using 4 CPU in my STAR-Fusion command and allocating a lot of memory to the job, I see that STAR-Fusion only uses a max of 1GB according to LSF logs. Just wondering if this expected and if there's any way i can improve the performance by maximizing memory usage.

[brianjohnhaas]

Hi,

Can you share your STAR-Fusion command?

Usually it requires a lot of RAM (like 40G), but if you're running it in quickstart mode (by just giving it the chimeric junction file), it might use a lot less RAM. In general, you make it faster by giving it a higher --CPU value. Not all parts are multithreaded though, but the slower parts should be for the most part.

On Thu, Aug 24, 2023 at 8:51 PM Anne Marie Noronha @.***> wrote:

Despite using 4 CPU in my STAR-Fusion command and allocating a lot of memory to the job, I see that STAR-Fusion only uses a max of 1GB according to LSF logs. Just wondering if this expected and if there's any way i can improve the performance by using more memory.

— Reply to this email directly, view it on GitHub https://github.com/STAR-Fusion/STAR-Fusion/issues/372, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX3VHE4LI6SZY34U56LXW7ZHXANCNFSM6AAAAAA35Y7NMM . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[anoronh4]

Thanks for your reply. Yes i am using "quick-start" mode:

STAR-Fusion \
    --genome_lib_dir ctat_genome_lib_build_dir \
    -J mySample.Chimeric.out.junction \
    --CPU 6 \
    --examine_coding_effect \
    --output_dir . 

I have a sample that has failed several times after running into time limits. This latest command has 6 CPU and went on for 12 hours before getting killed. hopefully my next run will do the trick...but are these run-times expected in certain cases? most of my jobs complete in <1 hours with only 2 cpu.

looking at the logs of samples with this issue i also see things like:

-mapping reads to genes
[3900000], rate=5442.62/min caught USR2/TERM signal

That seems like a really slow rate compared to my other jobs that finished much faster, but i'm also confused because there are no reads, just the junction file.

By the way i'm using version 1.10.1

[brianjohnhaas]

This read to gene mapping stage should be highly scalable with the --CPU setting. If it's slow for some reason, it could have to do with the file system, as it's very I/O intensive in terms of reading a very large data file and all threads are all hitting the file system separately. I don't think it should be memory intensive, though, since it's not running STAR and I don't think there's any steps beyond that requiring much in terms of RAM.

~b

On Fri, Aug 25, 2023 at 12:00 PM Anne Marie Noronha < @.***> wrote:

Thanks for your reply. Yes i am using "quick-start" mode:

STAR-Fusion \ --genome_lib_dir ctat_genome_lib_build_dir \ -J mySample.Chimeric.out.junction \ --CPU 6 \ --examine_coding_effect \ --output_dir . \

I have a sample that has failed several times after running into time limits. This latest command has 6 CPU and went on for 12 hours before getting killed. hopefully my next run will do the trick...but are these run-times expected in certain cases? most of my jobs complete in <1 hours with only 2 cpu.

looking at the logs i also see things like:

-mapping reads to genes [3900000], rate=5442.62/min caught USR2/TERM signal

That seems like a really slow rate compared to my other jobs that finished much faster, but i'm also confused because there are no reads, just the junction file.

— Reply to this email directly, view it on GitHub https://github.com/STAR-Fusion/STAR-Fusion/issues/372#issuecomment-1693582500, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX4TC6YWCSGVVTR5GC3XXDDYPANCNFSM6AAAAAA35Y7NMM . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[anoronh4]

I need to upgrade starfusion though, right? I'm seeing that for release v1.11.0 it says:

  - added multithreading for chimeric read mapping to genes for faster execution times.

am i interpreting that correctly, that this enhancement is directly improving the step where my job times out?

[brianjohnhaas]

Yes, if you're using an old version, definitely upgrade. I didn't realize you were using an old version here.

On Fri, Aug 25, 2023 at 1:00 PM Anne Marie Noronha @.***> wrote:

I need to upgrade starfusion though, right? I'm seeing that for release v1.11.0 it says:

• added multithreading for chimeric read mapping to genes for faster execution times.

am i interpreting that correctly, that this enhancement is directly improving the step where my job times out?

— Reply to this email directly, view it on GitHub https://github.com/STAR-Fusion/STAR-Fusion/issues/372#issuecomment-1693672411, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX46OHNHMWYFJGGNGOLXXDKZDANCNFSM6AAAAAA35Y7NMM . You are receiving this because you commented.Message ID: @.***>

--

Brian J. Haas The Broad Institute http://broadinstitute.org/~bhaas http://broad.mit.edu/~bhaas

[anoronh4]

no worries! i edited the comment after the fact (bad habit).

looks like the upgraded version let the job finish...much faster. thank you so much!

[brianjohnhaas]

I love to hear this!

On Fri, Aug 25, 2023 at 4:57 PM Anne Marie Noronha @.***> wrote:

no worries! i edited the comment after the fact (bad habit).

looks like the upgraded version let the job finish...much faster. thank you so much!

— Reply to this email directly, view it on GitHub https://github.com/STAR-Fusion/STAR-Fusion/issues/372#issuecomment-1693921665, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZRKX6ZLKP4J6PDIG7HZGLXXEGULANCNFSM6AAAAAA35Y7NMM . You are receiving this because you commented.Message ID: @.***>