rob-p
commented
7 years ago Hi @lcolladotor,
That's really strange. It is the case that it's possible for Salmon to use more than one thread if you set -p 1 --- it will use up to 2 threads in this case (1 is always dedicated for parsing). However, none of this would explain the memory usage you see. Adding more threads should not substantially change the memory burden in any way, as the main memory usage is in memory shared between threads and the amount of thread local memory is very small (a few MB at most). I would recommend updating to the latest version of Salmon (v0.8.2), as it does reduce the memory usage even more. I commonly quantify on the human transcriptome, and this uses 3-4G on my machine (with 16 threads). If you're still seeing this behavior with the newest version, we can try and debug further --- though I'm not sure exactly where to look next.
lcolladotor
Hi,
We (@emilyburke, @andrewejaffe and me) can successfully run salmon using 1 core in the SGE-managed cluster we have access to. The problem is that for some reason we have to request a lot of memory, thus reducing the number of jobs we can run simultaneously and increasing the overall time it takes to process datasets.
I see in https://github.com/COMBINE-lab/salmon/issues/97 that you suggest using
--perfectHashand a newersalmonversion, which we haven't tried and might end up being the solution.Here is the bash script we used to run
salmonfor a 422 sample dataset:The important part is that we are requesting 1 single core with 80 GB of free memory and setting the limit at 90 GB. We are also using the
-p 1option.Here is the log file for one of them (task 3):
The log looks ok to us.
Using
qacct(an SGE command) I can check the memory it used, and it used ~9 GB of RAM:For task 1 the maxvmem was 9.072G and for task 2 9.061G.
I then ran a test requesting a minimum of 10 GB of free RAM and a max of 11 GB, which in theory should work unless
salmonuses variable amounts of memory with the same data.The log file for the same file (task 3) now shows that there was a problem:
and the core dump file shows that the program was terminated:
The SGE information shows that it basically reached 10.7 GB of RAM:
I'm sure that the job got terminated because the memory reached the limit of 11 GB.
I previously did several tests where for a file the max memory reported was about 9 GB when requesting about 100G of RAM, and the same job kept failing even if I requested 10G, 20G, 30G, 40G... I didn't save the info then to report the problem.
Back on these tests, I then increased the memory requested a bit more (and used the
-m eSGE option to get an email with the max vmem, which matches theqacctoutput). Here is the bash script:and the log file for task 2:
as well as the
gdboutput for it's core dump file:and the SGE info:
The 80 GB memory free and 90 GB memory limit settings don't give us any of these issues as @emilyburke found out. So that's why we are using that crazy high memory setting. And for some reason in that case, the maximum memory used reported is around 9 GB...
Could it be that salmon is using more than 1 thread even though we are setting
-p 1? I know that our SGE cluster has some issues with shared memory across multiple threads.Best, Leo