serge2016
commented
4 years ago I suppose to replace | with or in line: https://github.com/etal/cnvkit/blob/master/cnvlib/smoothing.py#L197
This would remove warnings (but not the problem).
Open golubnikova opened 4 years ago
serge2016
commented
4 years ago I suppose to replace | with or in line: https://github.com/etal/cnvkit/blob/master/cnvlib/smoothing.py#L197
This would remove warnings (but not the problem).
Nash0990
commented
4 years ago Hi,
I encountered the same issue while running with amplicon method
cnvkit.py batch -m amplicon /Work/CNV/Validation/*bam -r CRR_Valid_31Samples.cnn -d ./results -p 30
CNVkit 0.9.7 Wrote ./results/CRR_Valid_31Samples.target-tmp.bed with 17249 regions Wrote ./results/CRR_Valid_31Samples.antitarget-tmp.bed with 0 regions Running 1 samples in 30 processes (that's 30 processes per bam) Running the CNVkit pipeline on /Work/CNV/Validation/KT40270255_CRR_VAL_deduped.bam ... Processing reads in KT40270255_CRR_VAL_deduped.bam Time: 8.747 seconds (263687 reads/sec, 1972 bins/sec) Summary: #bins=17249, #reads=2306593, mean=133.7233, min=0.0, max=752.5430463576159 Percent reads in regions: 60.909 (of 3786972 mapped) Wrote ./results/KT40270255_CRR_VAL_deduped.targetcoverage.cnn with 17249 regions Skip processing KT40270255_CRR_VAL_deduped.bam with empty regions file ./results/CRR_Valid_31Samples.antitarget-tmp.bed Wrote ./results/KT40270255_CRR_VAL_deduped.antitargetcoverage.cnn with 0 regions Processing target: KT40270255_CRR_VAL_deduped Keeping 16433 of 17249 bins Correcting for GC bias... Processing antitarget: KT40270255_CRR_VAL_deduped Wrote ./results/KT40270255_CRR_VAL_deduped.cnr with 16433 regions Segmenting ./results/KT40270255_CRR_VAL_deduped.cnr ... Segmenting with method 'cbs', significance threshold 0.0001, in 30 processes /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater bad_idx = (y > x.max()) | (y < x.min()) /Work/anaconda3/envs/CNVkit/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less bad_idx = (y > x.max()) | (y < x.min())
The tool stops without any other error or warnings.
Versions of my sessions are
pandas==1.0.4
numpy==1.18.5
Python 3.7.6
Seems this bug has been reported by multiple people already.
Please suggest as to whether something can be done to mitigate the issue.
Looking forward to your reply.
Thanks and Regards, Nahush BN
golubnikova
commented
4 years ago The problem with RuntimeWarning is solved for both tumor and germline analysis (wgs mode) after the following programms update:
pysam==0.15.4 -> pysam==0.16.0
numpy==1.18.4 -> numpy==1.18.5
Tumor analysis
Command:
/usr/local/bin/cnvkit.py batch /input/D2288/D2288_t.aligned.bam --normal /input/D2288/D2288_n.aligned.bam --seq-method wgs --segment-method cbs --processes 96 --output-dir /output/v097_somatic/new_output_try2/cnvkitRunDir --output-reference /output/v097_somatic/new_output_try2/cnvkitRunDir/reference.D2288.cnn --access /output/v097_somatic/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1/GRCh38.d1.vd1.bed
My log from tumor analysis
CNVkit 0.9.7
WGS protocol: recommend '--annotate' option (e.g. refFlat.txt) to help locate genes in output files.
Detected file format: bed
Estimated read length 151.0
Wrote /tmp/tmpm7ltmuca.bed with 100 regions
WGS average depth 253.82 --> using bin size 197
Splitting large targets
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.target.bed with 14851616 regions
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.antitarget.bed with 0 regions
Building a copy number reference from normal samples...
Processing reads in D2288_n.aligned.bam
Skip processing D2288_n.aligned.bam with empty regions file /output/v097_somatic/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.antitarget.bed
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_n.aligned.antitargetcoverage.cnn with 0 regions
Time: 3713.790 seconds (634048 reads/sec, 3999 bins/sec)
Summary: #bins=14851616, #reads=2354719245, mean=158.5497, min=0.0, max=82826.3774834437
Percent reads in regions: 84.309 (of 2792973206 mapped)
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_n.aligned.targetcoverage.cnn with 14851616 regions
No FASTA reference genome provided; skipping GC, RM calculations
Relative log2 coverage of chrX=-1.03, chrY=-1.01 (maleness=237 x 1.26 = 300) --> assuming male
Loading /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_n.aligned.targetcoverage.cnn
Loading /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_n.aligned.antitargetcoverage.cnn
Calculating average bin coverages
Calculating bin spreads
Targets: 203441 (1.3698%) bins failed filters (log2 < -5.0, log2 > 5.0, spread > 1.0)
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/reference.D2288.cnn with 14851616 regions
Running 1 samples in 96 processes (that's 96 processes per bam)
Running the CNVkit pipeline on /input/D2288/D2288_t.aligned.bam ...
Processing reads in D2288_t.aligned.bam
Time: 3297.294 seconds (301819 reads/sec, 4504 bins/sec)
Summary: #bins=14851616, #reads=995186578, mean=67.0086, min=0.0, max=83032.97350993377
Percent reads in regions: 44.902 (of 2216356401 mapped)
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.targetcoverage.cnn with 14851616 regions
Skip processing D2288_t.aligned.bam with empty regions file /output/v097_somatic/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.antitarget.bed
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.antitargetcoverage.cnn with 0 regions
Processing target: D2288_t.aligned
Keeping 14648175 of 14851616 bins
WARNING: Skipping correction for GC bias
Processing antitarget: D2288_t.aligned
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.cnr with 14648175 regions
Segmenting /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.cnr ...
Segmenting with method 'cbs', significance threshold 1e-06, in 96 processes
Post-processing /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.cns ...
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.cns with 81535 regions
Applying filter 'ci'
Filtered by 'ci' from 81535 to 51338 rows
Calling copy number with thresholds: -1.1 => 0, -0.25 => 1, 0.2 => 2, 0.7 => 3
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.call.cns with 51338 regions
Significant hits in 2341881/14648175 bins (16%)
Wrote /output/v097_somatic/new_output_try2/cnvkitRunDir/D2288_t.aligned.bintest.cns with 2341881 regionsGermline analysis
Command:
/usr/local/bin/cnvkit.py batch /input/dragen1/ESSE_03-1552_K1_S24.bam --normal --seq-method wgs --segment-method cbs --processes 96 --output-dir /output/v097_germ/new_output_try2/cnvkitRunDir --output-reference /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1/FlatReference.cnn --access /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1/GRCh38.d1.vd1.bed
My log from germline analysis:
CNVkit 0.9.7
WGS protocol: recommend '--annotate' option (e.g. refFlat.txt) to help locate genes in output files.
Detected file format: bed
Splitting large targets
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.target.bed with 585146 regions
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.antitarget.bed with 0 regions
Building a flat reference...
Detected file format: bed
No FASTA reference genome provided; skipping GC, RM calculations
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1/FlatReference.cnn with 585146 regions
Running 1 samples in 96 processes (that's 96 processes per bam)
Running the CNVkit pipeline on /input/dragen1/ESSE_03-1552_K1_S24.bam ...
Processing reads in ESSE_03-1552_K1_S24.bam
Time: 104.060 seconds (7596684 reads/sec, 5623 bins/sec)
Summary: #bins=585146, #reads=790514376, mean=1350.9695, min=0.0, max=227406.27814569537
Percent reads in regions: 90.093 (of 877440661 mapped)
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.targetcoverage.cnn with 585146 regions
Skip processing ESSE_03-1552_K1_S24.bam with empty regions file /output/v097_germ/new_output_try2/cnvkitRunDir/GRCh38.d1.vd1.antitarget.bed
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.antitargetcoverage.cnn with 0 regions
Processing target: ESSE_03-1552_K1_S24
Keeping 585146 of 585146 bins
WARNING: Skipping correction for GC bias
Processing antitarget: ESSE_03-1552_K1_S24
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.cnr with 585146 regions
Segmenting /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.cnr ...
Segmenting with method 'cbs', significance threshold 1e-06, in 96 processes
Smoothing overshot at 2 / 23676 indices: (-2.538143778852038, 2.397531426734699) vs. original (-5.35198817151136, 2.3120536919326953)
Post-processing /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.cns ...
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.cns with 12501 regions
Applying filter 'ci'
Filtered by 'ci' from 12501 to 6505 rows
Calling copy number with thresholds: -1.1 => 0, -0.25 => 1, 0.2 => 2, 0.7 => 3
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.call.cns with 6505 regions
Significant hits in 980/585146 bins (0.167%)
Wrote /output/v097_germ/new_output_try2/cnvkitRunDir/ESSE_03-1552_K1_S24.bintest.cns with 980 regions
Peterchungkachun
commented
4 years ago my command:
cnvkit.py batch K562IMS.sorted.bam -n bloodWGA.bam \
-m wgs -f hg38.fasta --annotate refFlat.txt -p 4 --access access-5k-mappable.hg38.bedmy packages pysam 0.16.0.1 numpy 1.18.5
Correcting for GC bias...
Loading ./bloodWGA.antitargetcoverage.cnn
Calculating average bin coverages
Calculating bin spreads
Targets: 3076 (1.7406%) bins failed filters (log2 < -5.0, log2 > 5.0, spread > 1.0)
Moved existing file ./reference.cnn -> ./reference.cnn.1
Wrote ./reference.cnn with 176724 regions
Running 1 samples in 4 processes (that's 4 processes per bam)
Running the CNVkit pipeline on K562IMS.sorted.bam ...
Processing reads in K562IMS.sorted.bam
Time: 89.651 seconds (395396 reads/sec, 1971 bins/sec)
Summary: #bins=176724, #reads=35447664, mean=200.5821, min=0.0, max=6158.946666666667
Percent reads in regions: 91.645 (of 38679480 mapped)
Wrote ./K562IMS.sorted.targetcoverage.cnn with 176724 regions
Skip processing K562IMS.sorted.bam with empty regions file ./access-5k-mappable.hg38.antitarget.bed
Wrote ./K562IMS.sorted.antitargetcoverage.cnn with 0 regions
Processing target: K562IMS.sorted
Keeping 173648 of 176724 bins
Correcting for GC bias...
Processing antitarget: K562IMS.sorted
Wrote ./K562IMS.sorted.cnr with 173648 regions
Segmenting ./K562IMS.sorted.cnr ...
Segmenting with method 'cbs', significance threshold 1e-06, in 4 processes
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in greater
bad_idx = (y > x.max()) | (y < x.min())
/home/peterchung/anaconda3/envs/cnv/lib/python3.7/site-packages/cnvlib/smoothing.py:197: RuntimeWarning: invalid value encountered in less
bad_idx = (y > x.max()) | (y < x.min())But it still has this error
Hi, @etal !
When I run CNVkit pipeline (v0.9.7) in wgs mode to make tumor analysis, command:
$CNVKIT batch /data/D2288/D2288_t.aligned.bam --normal /data/D2288/D2288_n.aligned.bam --seq-method wgs -p 96 --output-dir /output/test_test3 --output-reference /output/test_test3/ref/reference.cnn --fasta /ref/GRCh38.d1.vd1/GRCh38.d1.vd1.fa |& tee -a /output/test_test3/D2288.batch.logCNVkit fails with RunTimeWarning:
There is the same Warning in germline mode (wgs), but pipeline doesn't fail.
In issue #521 you supposed that the cause of the problem lies within pandas, numpy versions.
My current versions of the programms:
Pandas version (
pandas==1.0.4)Numpy version (
numpy==1.18.5)Python version is 3.7.5
Looking forward to your answer.
With best regards, Liliya