skoren
commented
3 years ago The proper memory resource for Canu should be consumable which means it gets reserved as soon as a job is scheduled on a node. That would avoid over-committing nodes even if hundreds of jobs run since each would have dedicated memory to it. Do you have such a memory option (qconf -sc |grep -i memory)? That would be the easiest solution.
When did you try to change the ovsMemory, was it after the store started building and it failed? That doesn't work since the sort jobs are configured at the very start of the store build process to fit into the memory. To change memory in the middle, you'd have to remove all the error_correct.ovlStore* files/folders. However, that would just create more jobs since each would have to be smaller. You can also limit the number of concurrent jobs launched by changing gridEngineArrayJobs="-t ARRAY_JOBS -tc 10" which should limit canu to at most 10 jobs. Leave ovsMemory=32.
AGI-chandler
We are using canu 2.1.1 on Linux with SGE:
canu -d Sikem_ONT_EC2 -p error_correct minReadLength=15000 genomeSize=5.4g corMaxEvidenceErate=0.15 corMhapFilterThreshold=0.0000000002 corMhapOptions="--threshold 0.80 --num-hashes 512 --num-min-matches 3 --ordered-sketch-size 1000 --ordered-kmer-size 14 --min-olap-length 5000 --repeat-idf-scale 50" mhapMemory=60g mhapBlockSize=500 ovlMerDistinct=0.975 useGrid=true gridEngineResourceOption="-pe smp THREADS -l mem_free=MEMORY" gridOptions="-V -q all.q -S /bin/bash" -nanopore Sikem-2002_*On the ovs step the processes are using too much memory and crashing compute nodes. I tried to add
ovsMemory=4gto the above command but it doesn't seem to be helping. Incanu.outfile I can see:-- Grid: ovs 4.000 GB 1 CPU (overlap store sorting)so it seems to be using the 4g there, but later in the file it says:
I can also see that
-M 29option is being passed to theovStoreSorterprogram. I think what is happening is that hundreds of jobs are being submitted at once and, since there is initially a lot of memory free on our compute nodes, they are all being run at once. Usually within a few minutes the node has crashed and needs to be reset. I think if-M 4could be used, then our nodes could handle it, or if jobs were started more slowly then the memory management of SGE would work...