search

wdecoster / NanoPlot

Plotting scripts for long read sequencing data
http://nanoplot.bioinf.be
MIT License
401 stars 48 forks source link

Resulting Image（png） Generation Issues #360

Open RADIOMUMM opened 3 months ago

RADIOMUMM commented 3 months ago

Hello,

qcpic1="%s/result/01.QC/%s_cleandata/LengthvsQualityScatterPlot_kde.png"%(output,samplename) qcpic2="%s/result/01.QC/%s_cleandata/Non_WeightedHistogramReadlength.png"%(output,samplename) qcpic3="%s/result/01.QC/%s_cleandata/Non_WeightedLogTransformed_HistogramReadlength.png"%(output,samplename) qcpic4="%s/result/01.QC/%s_cleandata/Yield_By_Length.png"%(output,samplename)

wdecoster commented 3 months ago

they don't generate the same results every time

Could you elaborate on which files you get and which you are missing?

RADIOMUMM commented 3 months ago

I used the command mentioned above and ran it twice in a row, using NanoFilt to intercept the Reads in between. (NanoPlot --> NanoFilt -->NanoPlot)

First run of NanoPlot (rawdata):

Second run of NanoPlot (cleandata)：

This doesn't happen every time; sometimes the results are the same for both runs.

wdecoster commented 3 months ago

I cannot think of any reason for that to happen.

RADIOMUMM commented 3 months ago

No Yield_By_Length.png file was generated this time

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_cleandata total 3208 -rw-r--r-- 1 root root 487796 Mar 26 08:22 LengthvsQualityScatterPlot_dot.html -rw-r--r-- 1 root root 44141 Mar 26 08:22 LengthvsQualityScatterPlot_dot.png -rw-r--r-- 1 root root 725753 Mar 26 08:22 LengthvsQualityScatterPlot_kde.html -rw-r--r-- 1 root root 63994 Mar 26 08:22 LengthvsQualityScatterPlot_kde.png -rw-r--r-- 1 root root 1484310 Mar 26 08:22 NanoPlot-report.html -rw-r--r-- 1 root root 3778 Mar 26 08:22 NanoPlot_20240326_0811.log -rw-r--r-- 1 root root 841 Mar 26 08:15 NanoStats.txt -rw-r--r-- 1 root root 844 Mar 26 08:15 NanoStats_post_filtering.txt -rw-r--r-- 1 root root 13250 Mar 26 08:16 Non_weightedHistogramReadlength.html -rw-r--r-- 1 root root 33020 Mar 26 08:16 Non_weightedHistogramReadlength.png -rw-r--r-- 1 root root 15029 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.html -rw-r--r-- 1 root root 47257 Mar 26 08:16 Non_weightedLogTransformed_HistogramReadlength.png -rw-r--r-- 1 root root 14333 Mar 26 08:15 WeightedHistogramReadlength.html -rw-r--r-- 1 root root 45382 Mar 26 08:15 WeightedHistogramReadlength.png -rw-r--r-- 1 root root 17818 Mar 26 08:15 WeightedLogTransformed_HistogramReadlength.html -rw-r--r-- 1 root root 43669 Mar 26 08:16 WeightedLogTransformed_HistogramReadlength.png -rw-r--r-- 1 root root 198366 Mar 26 08:16 Yield_By_Length.html -rw-r--r-- 1 root root 705 Mar 26 08:35 cyclone_qc.txt

(base)[root@418b066513c2 01.QC]# ll test_TB20009174_nocut_rawdata total 3196 -rw-r--r-- 1 root root 483754 Mar 26 08:00 LengthvsQualityScatterPlot_dot.html -rw-r--r-- 1 root root 48899 Mar 26 08:00 LengthvsQualityScatterPlot_dot.png -rw-r--r-- 1 root root 719694 Mar 26 08:00 LengthvsQualityScatterPlot_kde.html -rw-r--r-- 1 root root 84935 Mar 26 08:00 LengthvsQualityScatterPlot_kde.png -rw-r--r-- 1 root root 1451376 Mar 26 08:00 NanoPlot-report.html -rw-r--r-- 1 root root 3474 Mar 26 08:00 NanoPlot_20240326_0755.log -rw-r--r-- 1 root root 845 Mar 26 08:00 NanoStats.txt -rw-r--r-- 1 root root 13304 Mar 26 08:00 Non_weightedHistogramReadlength.html -rw-r--r-- 1 root root 32855 Mar 26 08:00 Non_weightedHistogramReadlength.png -rw-r--r-- 1 root root 15041 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.html -rw-r--r-- 1 root root 43907 Mar 26 08:00 Non_weightedLogTransformed_HistogramReadlength.png -rw-r--r-- 1 root root 14379 Mar 26 08:00 WeightedHistogramReadlength.html -rw-r--r-- 1 root root 41574 Mar 26 08:00 WeightedHistogramReadlength.png -rw-r--r-- 1 root root 17786 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.html -rw-r--r-- 1 root root 39531 Mar 26 08:00 WeightedLogTransformed_HistogramReadlength.png -rw-r--r-- 1 root root 177708 Mar 26 08:00 Yield_By_Length.html -rw-r--r-- 1 root root 41155 Mar 26 08:00 Yield_By_Length.png -rw-r--r-- 1 root root 707 Mar 26 08:35 cyclone_qc.txt

RADIOMUMM commented 3 months ago

I cannot think of any reason for that to happen. It's so weird.

微信截图_20240326172120

wdecoster commented 3 months ago

Can you share the log file of a run that failed to generate a png plot?

RADIOMUMM commented 3 months ago

Can you share the log file of a run that failed to generate a png plot?

2024-03-26 08:11:31,873 NanoPlot 1.42.0 started with arguments Namespace(N50=True, alength=False, bam=None, barcoded=False, color='#4CB391', colormap='Greens', cram=None, downsample=None, dpi=300, drop_outliers=False, fasta=None, fastq=['/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz'], fastq_minimal=None, fastq_rich=None, feather=None, font_scale=1, format=['png'], hide_stats=False, huge=False, info_in_report=False, legacy=None, listcolormaps=False, listcolors=False, loglength=False, maxlength=None, minlength=None, minqual=None, no_N50=False, no_static=False, no_supplementary=False, only_report=False, outdir='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata', path='/Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/', percentqual=False, pickle=None, plots=['kde', 'dot'], prefix='', raw=False, readtype='1D', runtime_until=None, store=False, summary=None, threads=20, title='test_TB20009174_nocut_cleandata', tsv_stats=False, ubam=None, verbose=False) 2024-03-26 08:11:31,873 Python version is: 3.8.18 (default, Sep 11 2023, 13:40:15) [GCC 11.2.0] 2024-03-26 08:11:31,885 Nanoget: Starting to collect statistics from plain fastq file. 2024-03-26 08:11:31,885 Nanoget: Decompressing gzipped fastq /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata_trimmed-reads.fastq.gz 2024-03-26 08:15:57,025 Reduced DataFrame memory usage from 4.4492950439453125Mb to 4.4492950439453125Mb 2024-03-26 08:15:57,069 Nanoget: Gathered all metrics of 291581 reads 2024-03-26 08:15:57,213 Calculated statistics 2024-03-26 08:15:57,213 Using sequenced read lengths for plotting. 2024-03-26 08:15:57,222 Removed 29 artefactual reads with very short length and very high quality. 2024-03-26 08:15:57,355 Calculated statistics 2024-03-26 08:15:57,356 NanoPlot: Valid color #4CB391. 2024-03-26 08:15:57,356 NanoPlot: Valid colormap Greens. 2024-03-26 08:15:57,381 NanoPlot: Creating length plots for Read length. 2024-03-26 08:15:57,381 NanoPlot: Using 291552 reads with read length N50 of 30095bp and maximum of 117406bp. 2024-03-26 08:15:59,533 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedHistogramReadlength as png (or png for --legacy) 2024-03-26 08:16:01,478 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/WeightedLogTransformed_HistogramReadlength as png (or png for --legacy) 2024-03-26 08:16:03,489 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedHistogramReadlength as png (or png for --legacy) 2024-03-26 08:16:05,567 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/Non_weightedLogTransformed_HistogramReadlength as png (or png for --legacy) 2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem: 2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught 2024-03-26 08:22:49,946 Created length plots 2024-03-26 08:22:49,952 NanoPlot: Creating Read lengths vs Average read quality plots using 291552 reads. 2024-03-26 08:22:52,388 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_dot as png (or png for --legacy) 2024-03-26 08:22:55,236 Saved /Project/result/test_TB20009174_nocut_Cyclone/result/01.QC/test_TB20009174_nocut_cleandata/LengthvsQualityScatterPlot_kde as png (or png for --legacy) 2024-03-26 08:22:55,236 Created LengthvsQual plot 2024-03-26 08:22:55,237 Writing html report. 2024-03-26 08:22:55,258 Finished!

wdecoster commented 3 months ago

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem: 2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

RADIOMUMM commented 3 months ago

This is it:

2024-03-26 08:22:49,945 No static plots are saved due to some kaleido problem: 2024-03-26 08:22:49,945 Transform failed with error code 1: Failed to serialize document: Uncaught

This doesn't help us, but for some reason it failed to create the PNG image for a particular plot. But you still have the plot in HTML format, right?

right

hoelzer commented 1 month ago

Hey, I have the same problem: no PNGs are produced and the errors are always:

2024-06-04 10:04:00,961 No static plots are saved due to some kaleido problem:
2024-06-04 10:04:00,961 Transform failed with error code 1: Failed to serialize document: Uncaught

I am running on a SLURM cluster and have NanoPlot installed in a Singularity container. NanoPlot version is 1.42.0

hoelzer commented 1 month ago

This might be related to

https://stackoverflow.com/questions/71118601/saving-a-plotly-image-not-working-with-kaleido-even-though-it-is-installed

I am still wondering if this has something to do with executing NanoPlot from a container and/or a HPC environment...

hoelzer commented 1 month ago

Further research revelad that this might be also a problem with internet connection that Kaleido apparently needs to convert HTML to PNG. THis would explain my issue bc I work on a quite restrictive HPC with a lot of ports closed.

See: https://github.com/plotly/Kaleido/issues/108

Maybe it is possible to also configure Kaleido in NanoPlot to work offline?