Overflow error in gene_recovery_heatmap

mossmatters commented 1 year ago

An error submitted to me by email using hybpiper gene_recovery_heatmap that the plot area was too wide. The script calculated it needed 87550x2300 pixels but apparently matplotlib has a limit of 65536 pixels. When the figure length was set manually to "400 inches" the heatmap was generated. The full error:

:

[INFO]:    HybPiper was called with these arguments:
           recovery_heatmap seq_lengths.tsv

[INFO]:    Number of samples in input lengths file is: 46
[INFO]:    Number of genes in input lengths file is: 1751
[INFO]:    figure_length: 583.67 inches, figure_height: 15.33 inches, sample_text_size: 10 points, gene_id_text_size: 10 points
[INFO]:    Saving heatmap as file "recovery_heatmap.png" at 150 DPI
Traceback (most recent call last):
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/bin/hybpiper", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/hybpiper/assemble.py", line 1836, in main
    args.func(args)
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/hybpiper/assemble.py", line 1634, in gene_recovery_heatmap_main
    gene_recovery_heatmap.main(args)
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/hybpiper/gene_recovery_heatmap.py", line 204, in main
    plt.savefig(f'{args.heatmap_filename}.{args.heatmap_filetype}', dpi=args.heatmap_dpi, bbox_inches='tight')
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/pyplot.py", line 1023, in savefig
    res = fig.savefig(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/figure.py", line 3343, in savefig
    self.canvas.print_figure(fname, **kwargs)
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backend_bases.py", line 2336, in print_figure
    renderer = _get_renderer(
               ^^^^^^^^^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backend_bases.py", line 1598, in _get_renderer
    print_method(io.BytesIO())
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backend_bases.py", line 2232, in <lambda>
    print_method = functools.wraps(meth)(lambda *args, **kwargs: meth(
                                                                 ^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backends/backend_agg.py", line 509, in print_png
    self._print_pil(filename_or_obj, "png", pil_kwargs, metadata)
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backends/backend_agg.py", line 457, in _print_pil
    FigureCanvasAgg.draw(self)
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backends/backend_agg.py", line 394, in draw
    self.renderer = self.get_renderer()
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/_api/deprecation.py", line 384, in wrapper
    return func(*inner_args, **inner_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backends/backend_agg.py", line 411, in get_renderer
    self.renderer = RendererAgg(w, h, self.figure.dpi)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/darenasv/miniconda3/envs/hybpiper2.1/lib/python3.11/site-packages/matplotlib/backends/backend_agg.py", line 84, in __init__
    self._renderer = _RendererAgg(int(width), int(height), dpi)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Image size of 87550x2300 pixels is too large. It must be less than 2^16 in each direction.

marianaherreraco commented 1 year ago

Hello, please could you help me? I have the same problem; even when I try to modify the --figure_length option this doesn't work. I have 2500 genes, and I need figure_length: 745.67 inches, (74566 x 3300 pixels) for my Heatmap. Can you guide me? What can I do to obtain the image?

chrisjackson-pellicle commented 1 year ago

Hi @mossmatters and @marianaherreraco,

Can you upload the seq_lengths.tsv files that produce these errors, please?

Cheers,

Chris

marianaherreraco commented 1 year ago

Hello, thank you for your answer. Unfortunately, I can not upload the file. I received this message "We don’t support that file type. Tray again with a GIF, JPEG, JPG, MOV, MP4, PNG, SVG, WEBM, CSV, DOCX, FODG, FODP, FODS, FODT, GZ, LOG, MD, ODF, ODG, ODP, ODS, ODT, PATCH, PDF, PPTX, TGZ, TXT, XLS, XLSX or ZIP." Styling with Markdown is supported.

chrisjackson-pellicle commented 1 year ago

@marianaherreraco are you able to zip the file, then upload it?

marianaherreraco commented 1 year ago

seq_lengths (2).zip Dear @chrisjackson-pellicle , here is the file. Thank you, for your response.

chrisjackson-pellicle commented 1 year ago

Hi @marianaherreraco,

Thanks for that!

I'll release an update for HybPiper that deals with this issue automatically, but for the moment you can produce a heatmap from your seq_lengths.tsv file using the command:

hybpiper recovery_heatmap seq_lengths.tsv  --heatmap_dpi 100 --figure_length 400

However, did you know that your seq_length.tsv file has a value of zero for all genes and samples? Did you perhaps run the hybpiper stats command from the wrong directory?

Cheers,

Chris

marianaherreraco commented 1 year ago

Hello @chrisjackson-pellicle Thank you, with your modification worked to generate the HeatMap. However, is true that using stats, the result hybpiper_stats (1).zip gave me Zero in all the seq_length, and for hybpiper_stats even when I have reads mapped, PctOnTarget and GenesMapped, I obtained zero GeneswithContigs (From genes with mapped reads, the number of genes for which SPAdes assembled contigs), so maybe is an issue the the assemble? I'm working in the same folder, should I modify the path to run the stats? Thank you for your support. Mariana

chrisjackson-pellicle commented 1 year ago

Hi @marianaherreraco,

Hmm, it looks like something went wrong with the SPAdes assemblies. Can you upload the sample log file for one of your samples (found in the sample directory, with a name like <prefix>_hybpiper_assemble_<date_time>.log), as well as any other relevant logs e.g. PBS/SLURM job logs?

Cheers,

Chris

chrisjackson-pellicle commented 1 year ago

@mossmatters @marianaherreraco - the heatmap issue should be resolved in HybPiper version 2.1.6, which has the changes:

Added flags --no_xlabels and --no_ylabels for hybpiper recovery_heatmap and hybpiper paralog_retriever; turns off rendering of the corresponding labels in the saved figures.
If the auto-calculated size of heatmaps for hybpiper recovery_heatmap and hybpiper paralog_retriever is greater than the maximum number of pixels (65536) in either/or length and height, resize the figure to 400 inches and 100 DPI. Note that large datasets can fail to render fully in the saved figure even if the pixel dimensions are less than the maximum (see e.g. https://stackoverflow.com/questions/64393779/how-to-render-a-heatmap-for-a-large-array), but reducing the size/DPI further allows the full figure to be rendered.

Cheers,

Chris

marianaherreraco commented 1 year ago

Dear @chrisjackson-pellicle thank you, here, I have included three files. Please, Can you check these logs? SPADES.zip Thank you for your support. M

chrisjackson-pellicle commented 1 year ago

Hi @marianaherreraco - thanks for those files.

The issue is caused by the quotation marks (") that are present in some of the gene names in your target file, e.g. PpGH49_019_g230350-"Alpha_1,2_mannosyltransferase"_g230350.

For HybPiper v2.1.7 I'll insert a check/warning for this, but in the meantime you can remove them using the following command:

sed 's/"//g' test_targets.fasta > test_targets_fixed.fasta

Then, re-run HybPiper using the test_targets_fixed.fasta target file.

Let me know how you go!

Cheers,

Chris

marianaherreraco commented 1 year ago

Thank you, yes, I did. Now everything runs pretty well. I had an issue with the paralogs, but I used --heatmap_dpi 100 --figure_length 400, and the run worked smoothly. Thank you for your quick answer. I appreciate your time and kindness. M

mossmatters / HybPiper

Overflow error in gene_recovery_heatmap #123