brianwalenz
commented
7 years ago That sounds a whole like the machines are out of memory. Can you log in and check swap usage and/or paging rate? While you're there, grab the output of top; I'd like to see memory usage of these jobs.
If they are paging, the correct solution is to decrease mhapMemory to 30gb or so, but this will change the layout of the jobs and you'll lose all previous work. Instead, increase mhapThreads to 13 to get only 3 jobs per node.
I can't think of what else could cause this. The mhap jobs aren't heavy consumers of I/O. They do a large data load at the start, then (possibly) stream inputs (these might be loaded at the start) and write outputs. All work is done in correction/1-overlapper, the data loaded comes from blocks/ (via symlinks in queries), and the results are written into results/. After the mhap java compute is done, results are rewritten into a format the rest of canu can understand.
moudixtc
skoren
Hi, I'm not sure if this question should be posted here, so please direct me to the correct place if needed.
I'm running Canu release v1.5 on a cluster of 5 AWS EC2 instances with ubuntu 14.04, each has 40 CPUs, and 160GB memory, and all connected to AWS EFS. The scheduler is slurm 16.05.3. The command I ran is basically the following but with different directories
canu -p maize_01DKD2 -d maize_01DKD2-auto genomeSize=2400m -pacbio-raw *.fastq.Here is the partial output from canu.*.out:
With the default slurm configuration, only one job can be running at the time on each node, so for my setup, 5 jobs are running simultaneously. I was at the
'mhap (cor)'stage, and each job took about 7 hours to finish. Not sure if that time sounds about right, but I felt it was slow, and since only one job is running on each node, about 75% of the node resource is not being used. As a result, I changed the slurm configuration to allow more than one job on each node. Basically fromSelectType=select/lineartoSelectType=select/cons_resandSelectTypeParameters=CR_CPU_Memory.Now I have 20 jobs running at the same time, which is expected, but the time each job took is significantly longer. As you can see from the following results, some jobs have been running for almost 5 days.
The CPU usage for these nodes also seems to be low, 3 of the 5 has an average of around 1%, while the other 2 are at about 5-10%.
Do you have any ideas on where the problem might be? Also you mentioned that canu works pretty well with slurm, so I'm also wondering if you have a recommended slurm configuration.