search

ENCODE-DCC / chip-seq-pipeline2

ENCODE ChIP-seq pipeline
MIT License
246 stars 123 forks source link

SGE qsub not requesting enough virtual memory to initialize Java virtual machine #132

Open xiaofanjin opened 4 years ago

xiaofanjin commented 4 years ago

Describe the bug Hi I am running the pipeline in an SGE environment, but running into issues at the encode_task_filter step with virtual memory.

Currently it seems the default memory request for encode_task_filter step is 20GB, and so the SGE qsub requests 4 cores with 5GB each of virtual memory (-pe smp 4 -l h_vmem=5GB -l s_vmem=5GB). However, to run picard in this task, the java virtual machine upon initialization will request 20GB for its heap, plus usually 1GB default for metaspace, on top of other requirements. As a result, this step ends up requiring more than 20GB in virtual memory, resulting in the error:

Error occurred during initialization of VM Could not reserve enough space for 20971520KB object heap

I have played around with this and increasing the qsub virtual memory request to 23GB seems likely to solve the problem. Less than that, but more than 20GB and the JVM has trouble allocating metaspace, resulting in something like:

Error occurred during initialization of VM Could not allocate metaspace: 1073741824 bytes

For now, adding the following line to the default.conf file fixes the issue, but its a little hacky sge-extra-param=-l h_vmem=23G -l s_vmem=23G

Would it be possible to modify the qsub -l h_vmem and -l s_vmem parameters in a future release so instead of requesting X/#threads of virtual memory, where X=memory-limit for a particular task, have qsub request something like (X+5GB)/#threads so there is a little extra space to initialize the JVM? Thanks

OS/Platform

Caper configuration file backend=sge sge-pe=smp

tmp-dir=/wynton/scratch

Input JSON file { "chip.pipeline_type" : "tf", "chip.genome_tsv" : "https://storage.googleapis.com/encode-pipeline-genome-data/genome_tsv/v1/hg38_chr19_chrM_caper.tsv", "chip.fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R1.subsampled.50.fastq.gz" ], "chip.fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep1-R2.subsampled.50.fastq.gz" ], "chip.fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R1.subsampled.50.fastq.gz" ], "chip.fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/rep2-R2.subsampled.50.fastq.gz" ], "chip.ctl_fastqs_rep1_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R1.subsampled.80.fastq.gz" ], "chip.ctl_fastqs_rep1_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl1-R2.subsampled.80.fastq.gz" ], "chip.ctl_fastqs_rep2_R1" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R1.subsampled.80.fastq.gz" ], "chip.ctl_fastqs_rep2_R2" : ["https://storage.googleapis.com/encode-pipeline-test-samples/encode-chip-seq-pipeline/ENCSR936XTK/fastq_subsampled/ctl2-R2.subsampled.80.fastq.gz" ], "chip.paired_end" : true, "chip.always_use_pooled_ctl" : true, "chip.title" : "ENCSR936XTK (subsampled 1/50, chr19 and chrM Only)", "chip.description" : "ZNF143 ChIP-seq on human GM12878" }

Error log errorlog.txt

ambeys commented 4 years ago

Hey!! how did it go finally. were u able to get it resolved?? I am in exactly same error. so just wandering if you can giude me please. thanks

xiaofanjin commented 4 years ago

If this is still an issue, try adding the following line to the ~/.caper/default.conf file: sge-extra-param=-l h_vmem=23G -l s_vmem=23G

ambeys commented 4 years ago

Right now its running with qsub lets see. if not will try. thanks a lot for responding.

leepc12 commented 4 years ago

Sorry for late response, I think I missed this issue.

The latest pipeline automatically finds required resources for a task based on input file sizes (BAM for filter task, FASTQs for align task, ...). Also, JAVA heap size (java -Xmx) is taken as 90% of task's memory.

Please try with the latest pipeline and let me know if it doesn't work.