search

broadinstitute / pilon

Pilon is an automated genome assembly improvement and variant detection tool
GNU General Public License v2.0
340 stars 60 forks source link

Pilon with MP + PE cuses too many cpus and memory #58

Closed jaworskicoline closed 5 years ago

jaworskicoline commented 7 years ago

Hello, I'm trying to use Pilon with both paired-ends and mate-pairs, and, as compared to paired-end only, it requires way more memory capacity, and it also uses more cpus than allowed, and thus gets killed because trying to use more cpus (29.20) than there is on one node (28). Here is the command: $ java -Xmx64G -jar ./pilon-1.22.jar --genome Assembly.fasta --frags ./Aligned_readsPE/Aligned_reads_sorted.bam --jumps ./Aligned_readsMP/Aligned_reads_sorted.bam --output Assembly_PilonPEMP --threads 24

My questions: 1/ Is it normal that it requires way more memory and cpus than with PE only ? 2/ how can I solve my issue (ie make it take into account the --threads command) ?

Thank you for your help ! Best regards, Coline

jaworskicoline commented 7 years ago

Hello again, I got it working ! Apparently, the problem was not really memory, but that it tried to use more cpus than given access to, and thus it caused memory issues because trying to run too many things at a time, and then time issue due to memory limitation. To solve that, I actually gave it a full node (28 cpus), but allowed it to run on less (eg, between 10 and 16, depending on the assembly, and time/memory needs) via the command line (--threads option). It actually used more than those 10 to 16, but the job was not killed because that remained below the 28.