search

broadinstitute / pilon

Pilon is an automated genome assembly improvement and variant detection tool
GNU General Public License v2.0
340 stars 60 forks source link

pilon on sge cluster #55

Closed dcopetti closed 5 years ago

dcopetti commented 7 years ago

Hi, I am running for the first time Pilon on our cluster (8 nodes, ~200 cores, ~600 GB RAM) java -Xmx600G -jar /opt/pilon-1.18.jar --genome Pdac_fu_124_pa_ctg_5188.fa --frags pdac_bwamem_s.bam --output Pdac_fu_124_pa_cns17988_pilon.fa --diploid --threads 180 --changes --debug --chunksize 1000000 with an alignment file obtained with bwa mem. It starts working, but I see that it is using only one node. After a while it crashes for too few RAM and I get the log file with written:

# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8674869248 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:2627), pid=9470, tid=139228591601408
#
# JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again

I am wondering how I should change my command to have Pilon using multiple nodes and cores. Thanks, Dario

w1bw commented 5 years ago

I see this never got a reponse. Pilon is not a distributed code by itself. You would have to split up the input genome passing different "--targets" as different jobs to run on the cluster.