Open AvivBenchorin opened 2 years ago
phbradley
commented
2 years ago Hi Aviv, Thanks for trying conga! I've run some really big sets and I haven't run into this yet, very cool! As a quick check, if you manually reduce the dpi in the plt.savefig command, say from 300 to 200 or 100, does that "fix" it (ie, make the error go away)? Also curious if you have any log output from before the error... Take care, Phil
AvivBenchorin
commented
2 years ago Hi Phil,
Here are the last couples lines of log output leading up to the error: Stdout:
making cluster logos: 603 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 604 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 605 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 606 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 607 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 608 610 CoNGA_out_restarted_tcr_clumping_logos.png
making cluster logos: 609 CoNGA_out_restarted_tcr_clumping_logos.png
making: CoNGA_out_restarted_tcr_clumping_logos.pngStderr:
.................................................. 80000
.................................................. 85000
.................................................. 90000
.................................................. 95000
.................................................. 100000
.................................................. 105000
.................................................. 110000
...................................
... storing 'test' as categorical
Traceback (most recent call last):
File "/path/to/conga/scripts/run_conga.py", line 831, in <module>
pvalue_threshold_for_logos=args.pvalue_threshold_for_tcr_clumping,
File "/path/to/conga/conga/plotting.py", line 3193, in make_tcr_clumping_plots
**logo_plot_args)
File "/path/to/conga/conga/plotting.py", line 3138, in make_cluster_logo_plots_figure
**kwargs)
File "/path/to/conga/conga/plotting.py", line 1450, in make_logo_plots
plt.savefig(logo_pngfile, dpi=300)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/pyplot.py", line 859, in savefig
res = fig.savefig(*args, **kwargs)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/figure.py", line 2311, in savefig
self.canvas.print_figure(fname, **kwargs)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 2217, in print_figure
**kwargs)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 1639, in wrapper
return func(*args, **kwargs)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 509, in print_png
FigureCanvasAgg.draw(self)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 402, in draw
self.renderer = self.get_renderer(cleared=True)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 418, in get_renderer
self.renderer = RendererAgg(w, h, self.figure.dpi)
File "/path/to/miniconda3/envs/conga_new_env/lib/python3.6/site-packages/matplotlib/backends/backend_agg.py", line 96, in __init__
self._renderer = _RendererAgg(int(width), int(height), dpi)
ValueError: Image size of 4124x92909 pixels is too large. It must be less than 2^16 in each direction.I will try to run the reanalyze step again with the reduced the DPI of 100. Do you have any suggestions for "fast-forwarding" to this specific step when running the run_conga.py command? With the size of the data, running the entire reanalyze step until the point that it previously crashed would take a few hours.
Best, Aviv
phbradley
commented
2 years ago Thanks! You could try replacing " --all " with " --tcr_clumping " but it will still take a little while, I'm afraid.
That is a lot of convergent TCR clusters! Another thing you could to to focus on the most interesting ones would be to add " --min_cluster_size_for_tcr_clumping_logos 10 "
The current default min size is 3, which may be too low for big datasets.
Are you expecting that degree of TCR sequence convergence? It makes me a tiny bit worried that somehow clonotypes might be getting "split" during the preprocessing, leading to apparent high sequence convergence (identical sequences shared between different clonotypes, but they are actually the same clonotype). Just a thought.
AvivBenchorin
commented
2 years ago Hi Phil,
Rerunning the reanalysis step after changing the DPI value to 100 in the make_logo_plots function made the error go away, although in the generated TCR clumping and graph_vs_graph visualization images the logos are a lower resolution, as would be expected, and slightly more difficult to read. As a workaround to get that part of the code running, however, it worked successfully, and I can play around with increasing the DPI values until I reach the DPI limit of having higher logo resolution and not breaking the savefig function call. Thank you!
The expected amount of TCR sequence convergence within the datasets is unknown, are there any easily modifiable parameters in the preprocessing steps could be tweaked to help reduce any potential "splitting" of clonotypes?
I ran into other issues that impeded on successfully running the rest of the CoNGA workflow, but I will open a new issue with the details of those problems.
Best, Aviv
When running the reanalyze step described in the README (running
run_conga.pywith--restartand--all), the run fails with the errorValueError: Image size of 4124x92909 pixels is too large. It must be less than 2^16 in each direction.while insidemake_tcr_clumping_plots.The error happens when
plt.savefig(logo_pngfile, dpi=300)is called inmake_logo_plots(line 1450 in conga/plotting.py), which is called bymake_cluster_logo_plots_figure(line 3130) and in turn is called inmake_tcr_clumping_plots(line 3189). It seems like the SVG to PNG conversion usingimagemagickwas successful for the individual logos plots, and the error is happening when merging the images together. Some quick Googling suggests that the issue may might be related toplt.textcalls inmake_logo_plots, but at the moment I am unsure of any specific causes.The commands I ran before the reanalyze step were:
The specific command I ran for the reanalyze step was:
python /path/to/conga/scripts/run_conga.py --restart CoNGA_out_final.h5ad --all --no_kpca --outfile_prefix CoNGA_out_restartedThe CoNGA run is being run on the gene expression and TCR data of 300k cells, using 10X cellranger outputs (the very large size of the data could be a contributing factor to the issue). I am using the most recent version of CoNGA (as of March 4, 2022, the most recent commit was December 10, 2021), and running CoNGA inside a conda python environment that was created using the instructions provided in the README.
My conda environment is the following: