search

bcbio / bcbio-nextgen

Validated, scalable, community developed variant calling, RNA-seq and small RNA analysis
https://bcbio-nextgen.readthedocs.io
MIT License
986 stars 354 forks source link

gatk-haplotype #427

Closed kspham closed 10 years ago

kspham commented 10 years ago

Hi Brad, I run bcbio-nextgen with -n 32 for gatk variant template. However, I see that the system consistently runs on only 2 CPUs !! Is it a bug or it's something that I don't understand. Thanks,

chapmanb commented 10 years ago

Son; Thanks for the report. It's tough to say for certain without more details about your system and what steps only use 2 CPUs. Is it the entire process or some specific component of the pipeline? Does the machine have enough memory to support 32 cores times the requested memory per core from the bcbio_system file? Is there any other details you can provide?

kspham commented 10 years ago

Hi Brad, This is the current state: (only one or 2 CPUs are used) ( I believe that this stage should be fully parallelized)

[2014-05-19 18:35] INFO 17:35:31,269 HelpFormatter - Program Args: -R /usr/local/share/bcbio-nextgen/genomes/Hsapiens/GRCh37/seq/GRCh37.fa --standard_min_confidence_threshold_for_calling 30.0 --standard_min_confidence_threshold_for_emitting 30.0 --downsample_to_coverage 10000 --downsampling_type BY_SAMPLE --annotation BaseQualityRankSumTest --annotation FisherStrand --annotation GCContent --annotation HaplotypeScore --annotation HomopolymerRun --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --annotation RMSMappingQuality --annotation DepthPerAlleleBySample --annotation Coverage -I /media/proj2/schi/hli_gatk_variant_csv/work/bamprep/1ns/5/1ns-ready-reorder-fixrgs-gatkfilter-5_26111483_28356956-prep.bam -I /media/proj2/schi/hli_gatk_variant_csv/work/bamprep/1s/5/1s-ready-reorder-fixrgs-gatkfilter-5_26111483_28356956-prep.bam --dbsnp /usr/local/share/bcbio-nextgen/genomes/Hsapiens/GRCh37/variation/dbsnp_138.vcf.gz -L /media/proj2/schi/hli_gatk_variant_csv/work/gatk-haplotype/5/gatk-5_26111483_28356956-raw-regions.bed --interval_set_rule INTERSECTION -T HaplotypeCaller -o /media/proj2/schi/hli_gatk_variant_csv/work/gatk-haplotype/5/tx/tmpUWuV1g/gatk-5_26111483_28356956-raw.vcf.gz --annotation ClippingRankSumTest --annotation DepthPerSampleHC -U LENIENT_VCF_PROCESSING --read_filter BadCigar --read_filter NotPrimaryAlignment

Memory is sufficient (128GB) resource configuration: log: dir: log ucsc_bigwig: memory: 36g bwa: cmd: bwa cores: 16 samtools: memory: 2G cores: 16 star: memory: 2g cores: 16 novoalign: cores: 16 memory: 2G gemini: cores: 16 freebayes: memory: 2g gatk: jvm_opts: ["-Xms7500m", "-Xmx25000m"] dir: /home/snow/bin/bcbio/tool/share/java/gatk gatk-haplotype: jvm_opts: ["-Xms2g", "-Xmx5500m"] gatk-vqsr: jvm_opts: ["-Xms2g", "-Xmx4000m"] picard: jvm_opts: ["-Xms750m", "-Xmx2500m"] dir: /home/snow/bin/bcbio/tool/share/java/picard snpeff: jvm_opts: ["-Xms750m", "-Xmx6g"] dir: /home/snow/bin/bcbio/tool/share/java/snpeff bcbio_variation: jvm_opts: ["-Xms750m", "-Xmx2500m"] dir: /home/snow/bin/bcbio/tool/share/java/bcbio_variation rnaseqc: dir: /home/snow/bin/bcbio/tool/share/java/RNA-SeQC jvm_opts: ["-Xms750m", "-Xmx20g"] mutect: jvm_opts: ["-Xms750m", "-Xmx2500m"] dir: /home/snow/bin/bcbio/tool/share/java/mutect varscan: jvm_opts: ["-Xms750m", "-Xmx2000m"] dir: /home/snow/bin/bcbio/tool/share/java/varscan oncofuse: jvm_opts: ["-Xms750m", "-Xmx2000m"] dir: /home/snow/bin/bcbio/tool/share/java/oncofuse AlienTrimmer: jvm_opts: ["-Xms750m", "-Xmx2000m"] dir: /home/snow/bin/bcbio/tool/share/java/AlienTrimmer

bowtie2: cores: 16 tophat2: cores: 16 cufflinks: cores: 16 bowtie: cores: 16 tophat: cores: 16

Location of galaxy configuration file, which has pointers to reference

data # https://bcbio-nextgen.readthedocs.org/en/latest/contents/configuration.html#reference-genome-files galaxy_config: universe_wsgi.ini

On Mon, May 19, 2014 at 6:34 PM, Brad Chapman notifications@github.comwrote:

Son; Thanks for the report. It's tough to say for certain without more details about your system and what steps only use 2 CPUs. Is it the entire process or some specific component of the pipeline? Does the machine have enough memory to support 32 cores times the requested memory per core from the bcbio_system file? Is there any other details you can provide?

— Reply to this email directly or view it on GitHubhttps://github.com/chapmanb/bcbio-nextgen/issues/427#issuecomment-43577900 .

chapmanb commented 10 years ago

Son; Your GATK memory specifications ask for 25GB of memory/core:

gatk: 
  jvm_opts: ["-Xms7500m", "-Xmx25000m"]

I'd still expect the pipeline to use 4 or 5 cores with this setup and your described total memory but definitely not 32. You probably want to reduce this to make better use of your cores and improve parallelization. Hope this helps.

kspham commented 10 years ago

Great -- thank you Brad, Do you know if I can stop and change the config and rerun it without deleting some directories ? Will the pipeline know the point where to restart? Thanks,

On Mon, May 19, 2014 at 7:21 PM, Brad Chapman notifications@github.comwrote:

Son; Your GATK memory specifications ask for 25GB of memory/core:

gatk: jvm_opts: ["-Xms7500m", "-Xmx25000m"]

I'd still expect the pipeline to use 4 or 5 cores with this setup and your described total memory but definitely not 32. You probably want to reduce this to make better use of your cores and improve parallelization. Hope this helps.

— Reply to this email directly or view it on GitHubhttps://github.com/chapmanb/bcbio-nextgen/issues/427#issuecomment-43580130 .

chapmanb commented 10 years ago

Son; Absolutely. It's safe to stop and restart at any point, and should pick up where it left off. Hope this gives you better parallelism.