search

mskcc / vcf2maf

Convert a VCF into a MAF, where each variant is annotated to only one of all possible gene isoforms
Other
374 stars 217 forks source link

(1.6.21) Not able to use all available CPU threads in the later part of VEP #316

Open sclan opened 2 years ago

sclan commented 2 years ago

The dockerized command and outputs, along with dstat lines (set one entry per 10 minutes during the run) on a 36 threaded computer. With "--vep-forks 45", the observation is ~23 cpu threads are used (it fluctuates) during the first 2 hours. Then it decreased to a single threaded (36 * 3% = 1) process and I wonder why. The VEP runs on the local VEP cache downloaded from Ensembl. 

$ perl /opt/vcf2maf.pl --input-vcf file.g.vcf --output-maf file.g.maf --ref-fasta ref.fasta --species homo_sapiens --ncbi-build GRCh37 --tmp-dir /tmp --vep-path /usr/local/bin --vep-data .vep --vep-forks 45

CPU: 1% (36 cores) * Memory: 12617/70235MB * Storage: 36/841GB * Net: 0↓/0↑MBps

STATUS: Running VEP and writing to: /tmp/file.g.vep.vcf
Possible precedence issue with control flow operator at /usr/local/lib/site_perl/5.26.2/Bio/DB/IndexedBase.pm line 805.

CPU: 38% (36 cores) * Memory: 16819/70235MB * Storage: 37/841GB * Net: 0↓/0↑MBps
CPU: 66% (36 cores) * Memory: 17682/70235MB * Storage: 40/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 18342/70235MB * Storage: 43/841GB * Net: 0↓/0↑MBps
CPU: 62% (36 cores) * Memory: 16353/70235MB * Storage: 46/841GB * Net: 0↓/0↑MBps
CPU: 63% (36 cores) * Memory: 19312/70235MB * Storage: 49/841GB * Net: 0↓/0↑MBps
CPU: 64% (36 cores) * Memory: 20737/70235MB * Storage: 52/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 16275/70235MB * Storage: 54/841GB * Net: 0↓/0↑MBps
CPU: 67% (36 cores) * Memory: 18305/70235MB * Storage: 57/841GB * Net: 0↓/0↑MBps
CPU: 65% (36 cores) * Memory: 16429/70235MB * Storage: 60/841GB * Net: 0↓/0↑MBps
CPU: 66% (36 cores) * Memory: 19054/70235MB * Storage: 63/841GB * Net: 0↓/0↑MBps
CPU: 68% (36 cores) * Memory: 20221/70235MB * Storage: 66/841GB * Net: 0↓/0↑MBps
CPU: 69% (36 cores) * Memory: 19972/70235MB * Storage: 69/841GB * Net: 0↓/0↑MBps
CPU: 69% (36 cores) * Memory: 20292/70235MB * Storage: 71/841GB * Net: 0↓/0↑MBps
WARNING: No genotype column for TUMOR in VCF!
WARNING: No genotype column for NORMAL in VCF!
CPU: 51% (36 cores) * Memory: 16459/70235MB * Storage: 73/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16445/70235MB * Storage: 74/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16431/70235MB * Storage: 74/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16419/70235MB * Storage: 75/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16403/70235MB * Storage: 76/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16377/70235MB * Storage: 76/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16404/70235MB * Storage: 77/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16365/70235MB * Storage: 77/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16381/70235MB * Storage: 78/841GB * Net: 0↓/0↑MBps
CPU: 3% (36 cores) * Memory: 16387/70235MB * Storage: 78/841GB * Net: 0↓/0↑MBps