Closed hyunjokoo closed 6 months ago
SamStudio8
commented
6 months ago Hi @hyunjokoo, we have an upcoming version that will allow finer control over the memory allocated to the annotation process. I will update this issue once it has been released. Thanks for your patience!
hyunjokoo
commented
6 months ago Is there any way we can setup memory allocation using java -Xms256m -Xmx4g etc.
I tested some, and found that when I use 48 Gb data, there was no error and the process was finished in 50 minutes, but when I used about 86 Gb data, the process was stopped with error. For now we plan to use the data from one Nanopore cell (about 90 Gb), but this workflow seems not allow such amount of data.
================================== I also tried in another server. This server uses Centos (128 cores -> 256 threads with hyperthreading, 2 Tb memory). When I tried the same data (48 Gb) which was ok with Ubuntu server (64 cores -> 128 threads with hyperthreading, 4 Tb memory), it failed in Centos server. The error memssage is like this. Do you think I should setup something in Centos server?
--------------------------------------------------------------------------------
This is epi2me-labs/wf-human-variation v1.9.1-gdfde912.
--------------------------------------------------------------------------------
executor > local (157)
[a9/8084b0] process > bam_ingress:check_for_alignment (1) [100%] 1 of 1 ✔
executor > local (210)
[a9/8084b0] process > bam_ingress:check_for_alignment (1) [100%] 1 of 1 ✔
executor > local (210)
[a9/8084b0] process > bam_ingress:check_for_alignment (1) [100%] 1 of 1 ✔
executor > local (210)
[a9/8084b0] process > bam_ingress:check_for_alignment (1) [100%] 1 of 1 ✔
[- ] process > bam_ingress:cram_to_bam -
[- ] process > bam_ingress:minimap2_alignment -
[d0/21be68] process > getGenome (1) [100%] 1 of 1 ✔
[4e/42e71d] process > cram_cache (1) [100%] 1 of 1 ✔
[18/675fa9] process > sanitise_bed (1) [100%] 1 of 1 ✔
[27/3f66bb] process > mosdepth_input (1) [100%] 1 of 1 ✔
[97/649c40] process > readStats (1) [ 0%] 0 of 1
[47/5c642f] process > getVersions [100%] 1 of 1 ✔
[06/03f563] process > getParams [100%] 1 of 1 ✔
[4d/387f80] process > get_region_coverage (1) [100%] 1 of 1 ✔
[- ] process > makeAlignmentReport -
[- ] process > failedQCReport -
[c0/ef96e1] process > lookup_clair3_model (1) [100%] 1 of 1 ✔
[5f/9f8f52] process > snp:make_chunks (1) [100%] 1 of 1 ✔
[86/721aad] process > snp:pileup_variants (254) [ 0%] 1 of 696
[- ] process > snp:aggregate_pileup_variants -
[- ] process > snp:select_het_snps -
[- ] process > snp:phase_contig -
[- ] process > snp:get_qual_filter -
[- ] process > snp:create_candidates -
[- ] process > snp:evaluate_candidates -
[- ] process > snp:aggregate_full_align_variants -
[- ] process > snp:merge_pileup_and_full_vars -
[- ] process > snp:aggregate_all_variants -
[a2/9f6248] process > sv:variantCall:filterBam (1) [ 0%] 0 of 1
[- ] process > sv:variantCall:sniffles2 -
[- ] process > sv:variantCall:filterCalls -
[- ] process > sv:variantCall:sortVCF -
[- ] process > sv:annotate_sv_vcf -
[0f/8f47e3] process > sv:runReport:getVersions [100%] 1 of 1 ✔
[1b/7b4d39] process > sv:runReport:getParams [100%] 1 of 1 ✔
[- ] process > sv:runReport:report -
[- ] process > output_sv -
[- ] process > refine_with_sv -
[- ] process > bed_filter -
[- ] process > annotate_snp_vcf -
[- ] process > vcfStats -
[97/5a341e] process > report_snp:getVersions [100%] 1 of 1 ✔
[e6/4b55b5] process > report_snp:getParams [100%] 1 of 1 ✔
[- ] process > report_snp:makeReport -
[- ] process > output_snp -
[cb/a144ad] process > configure_jbrowse (1) [100%] 1 of 1 ✔
[a4/ea52a5] process > publish_artifact (9) [100%] 9 of 9
ERROR ~ Execution aborted due to an unexpected error
-- Check '.nextflow.log' file for details
ERROR ~ Execution aborted due to an unexpected error
-- Check '.nextflow.log' file for details
[363.634s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[363.635s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HandshakeCompletedNotify-Thread"
[365.300s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[365.301s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HandshakeCompletedNotify-Thread"
ERROR ~ java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
-- Check script '/home/hjk/.nextflow/assets/epi2me-labs/wf-human-variation/main.nf' at line: 663 or see '.nextflow.log' file for more details
[365.316s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
[365.317s][warning][os,thread] Failed to start the native thread for java.lang.Thread "Thread-0"Thank you. HJK
hyunjokoo
commented
6 months ago Before the run of wf-human-variation process, I ran export _JAVA_OPTIONS=-Xmx1000g.
When the process starts, it prints the following line showing that maximum java heap memory is adjusted correctly as I entered.
Picked up _JAVA_OPTIONS: -Xmx1000g
But, the process failed in the java process again.
I found that the java process use only 4 Gb memory, and my previous setting is useless.
When I checked using top -c, I saw that it set up the maximum heap memory as 4 Gb.
java -Xmx4g -jar /home/epi2melabs/conda/share/snpeff-5.1-2/snpEff.jar ann GRCh38.p13 input.vcf.gz
I think you can simply add option for this process, or adjust the parameters. For example, if JAVA_OPTIONS were picked up, do not add -Xmx4g in this process, etc.
I hope this issue can be solved quickly.
Thank you. HJK
P.S. After downloading snpEff.jar, I executed the following in the corresponding workspace folder, and the process was finished in 3 hrs without error. But I do not know how to incorporate the result to the overall process.
[hjk@ngs9:/data_ngs10/hjk/data/2023/nanopore_epi2mewf-human-variation_test_human_minion/epi2me-human-variation4_NGS9/workspace/24/9a8899bc20cd2a751a0ce037a46db8]$ java -Xmx1000g -jar /home/hjk/utils/SnpEff/snpEff/snpEff.jar ann GRCh38.p13 input.vcf.gz
SamStudio8
commented
6 months ago Hi @hyunjokoo, thanks for raising this. We have fixed this internally as part of work for our next release which overhauls the CPU and Memory directives.
On Thu, 14 Dec 2023, 00:08 hyunjokoo, @.***> wrote:
Before the run of wf-human-variation process, I ran export _JAVA_OPTIONS=-Xmx1000g. When the process starts, it prints the following line showing that maximum java heap memory is adjusted correctly as I entered. Picked up _JAVA_OPTIONS: -Xmx1000g But, the process failed in the java process again.
I found that the java process use only 4 Gb memory, and my previous setting is useless. When I checked using top -c, I saw that it set up the maximum heap memory as 4 Gb. java -Xmx4g -jar /home/epi2melabs/conda/share/snpeff-5.1-2/snpEff.jar ann GRCh38.p13 input.vcf.gz
I think you can simply add option for this process, or adjust the parameters. For example, if JAVA_OPTIONS were picked up, do not add -Xmx4g in this process, etc.
I hope this issue can be solved quickly.
Thank you. HJK
— Reply to this email directly, view it on GitHub https://github.com/epi2me-labs/wf-human-variation/issues/125#issuecomment-1854882867, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIN6OSNDY5LWMM4EVVBXO3YJI7P5AVCNFSM6AAAAABAL4SYC2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNJUHA4DEOBWG4 . You are receiving this because you commented.Message ID: @.***>
hyunjokoo
commented
6 months ago hank you. I hope the next version comes quickly. ^^
Ask away!
Hello,
I downloaded human Nanopore data from NCBI. This MinION data is 79.72 Gb. https://www.ncbi.nlm.nih.gov/sra?linkname=bioproject_sra_all&from_uid=553438
I searched this website and found that this human genome is recommended. So I used this sequence as a reference.
ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gzUsing the query sequence and reference sequence, I did minimap2.
I also generated bed file for reference as shown below.
I started wf-human-variation workflow like this.
~/utils/nextflow/nextflow run epi2me-labs/wf-human-variation -w epi2me-human-variation3_NGS9/workspace -profile standard -with-docker --bam DRR180427_minimap2_to_GRCh38_sorted.bam --ref /data_ngs3/hjk/DB/human_genome_GRCh38_for_wf-human-variation/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna --sample_name DRR180427_human_MinION --snp --sv --out_dir epi2me-human-variation3_NGS9 --threads 127 --bed /data_ngs3/hjk/DB/human_genome_GRCh38_for_wf-human-variation/GCA_000001405.15_GRCh38_no_alt_analysis_set.bedI got error like this.
Could you please tell me what is the problem? How can I avoid this kind of error? This server has 64 cores (128 with hyperthreading) and 4 Tb memory.
I attach trace.txt file here. trace.txt
Thank you.
HJK