search

fritzsedlazeck / Spectre

Copy number caller for long read data including SNV utilization
MIT License
54 stars 3 forks source link

No .vcf.gz/.vcf.gz.tbi file generated when no variants are reported #35

Closed SergeWielhouwer closed 2 months ago

SergeWielhouwer commented 2 months ago

Hi,

I am currently still trying to set Spectre up correctly and I encountered a strange issue when using extremely low-coverage input data (mainly for quickly testing the Snakemake workflow then to really perform accurate CNV calling). When no variants are reported by Spectre, the program doesn't seem to write any .vcf.gz/.vcf.gz.tbi file or _cnv.bed.gz/_cnv.bed.gz.tbi file to the output directory. Ideally, I would expect an empty VCF file, then just no file at all as Snakemake (or other workflow managers) check for these output files.

What I expect as output.

What I got instead.

Any idea what's causing this?

Also the --output-file parameter, which is required for the RemoveNs function, seems to be missing from your readme (which also has a small typo). image

spectre::2024-09-27 17:01:37,677::INFO::spectre.spectre> Spectre input: CNVCaller --coverage variants/123456-001-003/spectre/mosdepth/123456-001-003.regions.bed.gz --sample-id 123456-001-003 --output-dir variants/123456-001-003/spectre/ --reference /mnt/shared/genomes/GRCh38/ncbi/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna --metadata /mnt/shared/tools/spectre-0.2.1/data/grch38.mdr --blacklist /mnt/shared/tools/spectre-0.2.1/data/grch38_blacklist_spectre.bed spectre::2024-09-27 17:01:37,677::INFO::spectre.spectre> Spectre version: 0.2.1 spectre::2024-09-27 17:01:37,677::INFO::spectre.spectre> Starting spectre spectre::2024-09-27 17:01:37,677::INFO::spectre.spectre> Evaluate if a new .mdr file needs to be created spectre::2024-09-27 17:01:37,678::INFO::spectre.spectre> Using existing metadata file /mnt/shared/tools/spectre-0.2.1/data/grch38.mdr spectre::2024-09-27 17:01:37,683::INFO::spectre.spectreCNV> Spectre calculating for: variants/123456-001-003/spectre/mosdepth/123456-001-003.regions.bed.gz spectre::2024-09-27 17:01:37,684::INFO::spectre.spectreCNV> Data normalization and outlier removal (right tail) [W::hts_idx_load3] The index file is older than the data file: variants/123456-001-003/spectre/mosdepth/123456-001-003.regions.bed.gz.csi [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? spectre::2024-09-27 17:01:38,254::INFO::spectre.analysis.analysis> Determined bin size from Mosdepth coverage: 1000 spectre::2024-09-27 17:01:38,259::WARNING::spectre.analysis.analysis> Minimum CNV length is smaller then 1000000 base pairs spectre::2024-09-27 17:01:38,259::WARNING::spectre.analysis.analysis> Expect trouble! [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? [W::tbx_parse1] Coordinate <= 0 detected. Did you forget to use the -0 option? spectre::2024-09-27 17:01:42,490::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr1: 248957 /usr/local/lib/python3.10/site-packages/spectre/analysis/analysis.py:272: RuntimeWarning: divide by zero encountered in divide normalized_candidates = self.coverage / self.normalization_value /usr/local/lib/python3.10/site-packages/spectre/analysis/analysis.py:272: RuntimeWarning: invalid value encountered in divide normalized_candidates = self.coverage / self.normalization_value /usr/local/lib/python3.10/site-packages/numpy/lib/nanfunctions.py:1734: RuntimeWarning: invalid value encountered in subtract np.subtract(arr, avg, out=arr, casting='unsafe', where=where) spectre::2024-09-27 17:01:42,606::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr2: 242194 spectre::2024-09-27 17:01:42,702::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr3: 198296 spectre::2024-09-27 17:01:42,779::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr4: 190215 spectre::2024-09-27 17:01:42,854::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr5: 181539 spectre::2024-09-27 17:01:42,926::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr6: 170806 spectre::2024-09-27 17:01:42,996::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr7: 159346 spectre::2024-09-27 17:01:43,061::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr8: 145139 spectre::2024-09-27 17:01:43,120::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr9: 138395 spectre::2024-09-27 17:01:43,179::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr10: 133798 spectre::2024-09-27 17:01:43,234::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr11: 135087 spectre::2024-09-27 17:01:43,289::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr12: 133276 spectre::2024-09-27 17:01:43,342::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr13: 114365 spectre::2024-09-27 17:01:43,390::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr14: 107044 spectre::2024-09-27 17:01:43,435::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr15: 101992 spectre::2024-09-27 17:01:43,479::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr16: 90339 spectre::2024-09-27 17:01:43,518::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr17: 83258 spectre::2024-09-27 17:01:43,553::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr18: 80374 spectre::2024-09-27 17:01:43,586::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr19: 58618 spectre::2024-09-27 17:01:43,610::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr20: 64445 spectre::2024-09-27 17:01:43,637::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr21: 46710 spectre::2024-09-27 17:01:43,657::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr22: 50819 spectre::2024-09-27 17:01:43,680::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chrX: 156041 spectre::2024-09-27 17:01:43,750::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chrY: 57228 spectre::2024-09-27 17:01:43,776::WARNING::spectre.analysis.analysis> Chromosome chrY: no data available! spectre::2024-09-27 17:01:43,776::INFO::spectre.analysis.analysis> Number positions to be tested on chromosome chr16_KI270728v1_random: 1873 spectre::2024-09-27 17:02:04,362::INFO::spectre.analysis.analysis> Selected threshold for DEL=nan and DUP=nan for a ploidy of 2 spectre::2024-09-27 17:02:04,402::INFO::spectre.spectreCNV> CNV calling - Coverage for sample: 123456-001-003 spectre::2024-09-27 17:02:04,402::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr1 spectre::2024-09-27 17:02:04,769::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr2 spectre::2024-09-27 17:02:05,065::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr3 spectre::2024-09-27 17:02:05,308::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr4 spectre::2024-09-27 17:02:05,542::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr5 spectre::2024-09-27 17:02:05,765::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr6 spectre::2024-09-27 17:02:05,974::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr7 spectre::2024-09-27 17:02:06,171::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr8 spectre::2024-09-27 17:02:06,351::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr9 spectre::2024-09-27 17:02:06,522::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr10 spectre::2024-09-27 17:02:06,687::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr11 spectre::2024-09-27 17:02:06,854::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr12 spectre::2024-09-27 17:02:07,019::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr13 spectre::2024-09-27 17:02:07,160::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr14 spectre::2024-09-27 17:02:07,294::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr15 spectre::2024-09-27 17:02:07,419::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr16 spectre::2024-09-27 17:02:07,531::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr17 spectre::2024-09-27 17:02:07,633::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr18 spectre::2024-09-27 17:02:07,732::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr19 spectre::2024-09-27 17:02:07,805::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr20 spectre::2024-09-27 17:02:07,885::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr21 spectre::2024-09-27 17:02:07,943::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr22 spectre::2024-09-27 17:02:08,006::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chrX spectre::2024-09-27 17:02:08,200::INFO::spectre.analysis.analysis> Calculating CNVs for 123456-001-003 @ chromosome chr16_KI270728v1_random spectre::2024-09-27 17:02:28,483::INFO::spectre.analysis.cnv_metrics> DEL border:nan DUP border:nan spectre::2024-09-27 17:02:28,483::INFO::spectre.analysis.analysis> refining cnv calls spectre::2024-09-27 17:02:28,485::INFO::spectre.spectreCNV> Calculate CNV metrics spectre::2024-09-27 17:02:48,863::INFO::spectre.analysis.cnv_metrics> DEL border:nan DUP border:nan spectre::2024-09-27 17:02:48,863::INFO::spectre.spectreCNV> Merge CNV and LoH candidates spectre::2024-09-27 17:02:48,864::DEBUG::spectre.classes.loh_candidate> CNV cand: 0 spectre::2024-09-27 17:02:48,864::INFO::spectre.spectreCNV> Final candidates are written to .spc file spectre::2024-09-27 17:02:48,870::INFO::spectre.spectreCNV> Results are writen to bed file spectre::2024-09-27 17:02:48,887::INFO::spectre.spectreCNV> Results are writen to VCF file spectre::2024-09-27 17:02:48,890::INFO::spectre.spectreCNV> Result plot in progress spectre::2024-09-27 17:02:49,331::INFO::spectre.plots.plot> CNVs in chromosome: chr1 spectre::2024-09-27 17:02:49,523::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr1.png spectre::2024-09-27 17:02:49,897::INFO::spectre.plots.plot> CNVs in chromosome: chr2 spectre::2024-09-27 17:02:50,035::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr2.png spectre::2024-09-27 17:02:50,351::INFO::spectre.plots.plot> CNVs in chromosome: chr3 spectre::2024-09-27 17:02:50,494::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr3.png spectre::2024-09-27 17:02:50,800::INFO::spectre.plots.plot> CNVs in chromosome: chr4 spectre::2024-09-27 17:02:50,941::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr4.png spectre::2024-09-27 17:02:51,303::INFO::spectre.plots.plot> CNVs in chromosome: chr5 spectre::2024-09-27 17:02:51,444::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr5.png spectre::2024-09-27 17:02:51,721::INFO::spectre.plots.plot> CNVs in chromosome: chr6 spectre::2024-09-27 17:02:51,860::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr6.png spectre::2024-09-27 17:02:52,125::INFO::spectre.plots.plot> CNVs in chromosome: chr7 spectre::2024-09-27 17:02:52,268::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr7.png spectre::2024-09-27 17:02:52,516::INFO::spectre.plots.plot> CNVs in chromosome: chr8 spectre::2024-09-27 17:02:52,657::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr8.png spectre::2024-09-27 17:02:52,894::INFO::spectre.plots.plot> CNVs in chromosome: chr9 spectre::2024-09-27 17:02:53,035::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr9.png spectre::2024-09-27 17:02:53,267::INFO::spectre.plots.plot> CNVs in chromosome: chr10 spectre::2024-09-27 17:02:53,409::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr10.png spectre::2024-09-27 17:02:53,643::INFO::spectre.plots.plot> CNVs in chromosome: chr11 spectre::2024-09-27 17:02:53,785::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr11.png spectre::2024-09-27 17:02:54,016::INFO::spectre.plots.plot> CNVs in chromosome: chr12 spectre::2024-09-27 17:02:54,155::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr12.png spectre::2024-09-27 17:02:54,360::INFO::spectre.plots.plot> CNVs in chromosome: chr13 spectre::2024-09-27 17:02:54,500::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr13.png spectre::2024-09-27 17:02:54,777::INFO::spectre.plots.plot> CNVs in chromosome: chr14 spectre::2024-09-27 17:02:54,917::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr14.png spectre::2024-09-27 17:02:55,108::INFO::spectre.plots.plot> CNVs in chromosome: chr15 spectre::2024-09-27 17:02:55,244::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr15.png spectre::2024-09-27 17:02:55,419::INFO::spectre.plots.plot> CNVs in chromosome: chr16 spectre::2024-09-27 17:02:55,553::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr16.png spectre::2024-09-27 17:02:55,719::INFO::spectre.plots.plot> CNVs in chromosome: chr17 spectre::2024-09-27 17:02:55,855::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr17.png spectre::2024-09-27 17:02:56,017::INFO::spectre.plots.plot> CNVs in chromosome: chr18 spectre::2024-09-27 17:02:56,160::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr18.png spectre::2024-09-27 17:02:56,294::INFO::spectre.plots.plot> CNVs in chromosome: chr19 spectre::2024-09-27 17:02:56,430::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr19.png spectre::2024-09-27 17:02:56,570::INFO::spectre.plots.plot> CNVs in chromosome: chr20 spectre::2024-09-27 17:02:56,709::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr20.png /usr/local/lib/python3.10/site-packages/spectre/plots/plot.py:39: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (matplotlib.pyplot.figure) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam figure.max_open_warning). Consider using matplotlib.pyplot.close(). self.figure = plot_engine.figure(figsize=(8, 4)) spectre::2024-09-27 17:02:56,828::INFO::spectre.plots.plot> CNVs in chromosome: chr21 spectre::2024-09-27 17:02:56,965::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr21.png spectre::2024-09-27 17:02:57,168::INFO::spectre.plots.plot> CNVs in chromosome: chr22 spectre::2024-09-27 17:02:57,305::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr22.png spectre::2024-09-27 17:02:57,566::INFO::spectre.plots.plot> CNVs in chromosome: chrX spectre::2024-09-27 17:02:57,707::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chrX.png spectre::2024-09-27 17:02:57,765::INFO::spectre.plots.plot> CNVs in chromosome: chr16_KI270728v1_random spectre::2024-09-27 17:02:57,907::INFO::spectre.plots.plot> Plot saved: img/123456-001-003_plot_cnv_chr16_KI270728v1_random.png spectre::2024-09-27 17:02:57,942::INFO::spectre.spectreCNV> Output dir: /mnt/scratch2/s.wielhouwer/ONT_variant_calling/pipeline/test_run_reporting/variants/123456-001-003/spectre spectre::2024-09-27 17:02:57,942::INFO::spectre.spectreCNV> Done with sample 123456-001-003 spectre::2024-09-27 17:02:57,942::INFO::spectre.spectre> Spectre finished

Thanks again for your input!

Serge

philippesanio commented 2 months ago

Hi @SergeWielhouwer

I am currently still trying to set Spectre up correctly and I encountered a strange issue when using extremely low-coverage input data (mainly for quickly testing the Snakemake workflow then to really perform accurate CNV calling). When no variants are reported by Spectre, the program doesn't seem to write any .vcf.gz/.vcf.gz.tbi file or _cnv.bed.gz/_cnv.bed.gz.tbi file to the output directory. Ideally, I would expect an empty VCF file, then just no file at all as Snakemake (or other workflow managers) check for these output files.

It seems you are not running the latest version of Spectre. Your version: 0.2.1. The latest public Spectre version 0.2.1-patch-july15 should address the issues. For the latest bug fixes, please always check GitHub. We will push the bug fixes with the next Spectre version to pip as well.

To install Spectre from GitHub locally using pip you can use the following commands:

pip install build
git clone https://github.com/fritzsedlazeck/Spectre.git
cd ./Spectre
python3 -m build
pip install dist/spectre_cnv-<VERSION>.tar.gz # replace <VERSION> with e.g. 0.2.0

Also the --output-file parameter, which is required for the RemoveNs function, seems to be missing from your readme (which also has a small typo).

Thank you for pointing that out! I will fix that asap!

Please let me know if that did the trick for you or if you are encountering any other problems.

Best, Philippe

SergeWielhouwer commented 2 months ago

Hi @philippesanio,

Thanks for pointing out the new patches and the quick fixes on the readme. I was checking out the "releases" section and did not realise that cloning from GIT directly would be a better approach. It is nice to see that this issue was already pointed out before and fixed accordingly.

I will try out installing the newest patch release and will get back to you if I encounter any new issues

Thanks again!

Serge