No heatmap shown with `seaborn` and `%matplotlib widget`

dominiquesydow commented 3 years ago

Describe the issue

Thank you very much for providing this tool!

I have an apparently seaborn-related problem with showing widgets; I am posting this here first. Please let me know if you think this problem is rather on seaborn's end and I'll repost it there.

The problem: seaborn figures are not shown in JupyterLab with %matplotlib widget (they do show up with %matplotlib inline):

%matplotlib widget
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

df = pd.DataFrame([[10, 20, 30, 40], [50, 30, 8, 15],
                   [25, 14, 41, 8], [7, 14, 21, 28]])
sns.heatmap(df, cmap='RdYlGn', linewidths=0.30, annot=True)

%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

df = pd.DataFrame([[10, 20, 30, 40], [50, 30, 8, 15],
                   [25, 14, 41, 8], [7, 14, 21, 28]])
sns.heatmap(df, cmap='RdYlGn', linewidths=0.30, annot=True)

No problem with matplotlib:

%matplotlib widget
import matplotlib.pyplot as plt

plt.plot([[1, 2], [0, 0]])

Versions

ipympl version: 0.8.2
Selected Jupyter core packages...
IPython          : 7.29.0
ipykernel        : 6.4.2
ipywidgets       : 7.6.5
jupyter_client   : 7.0.6
jupyter_core     : 4.9.1
jupyter_server   : 1.11.2
jupyterlab       : 3.2.2
nbclient         : 0.5.5
nbconvert        : 6.2.0
nbformat         : 5.1.3
notebook         : 6.4.5
qtconsole        : not installed
traitlets        : 5.1.1
Known nbextensions:
  config dir: /home/dominique/.jupyter/nbconfig
    notebook section
      nbextensions_configurator/config_menu/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/config_menu/main
      contrib_nbextensions_help_item/main  enabled 
      - Validating: OK
      spellchecker/main  enabled 
      - Validating: OK
      gist_it/main  enabled 
      - Validating: OK
    tree section
      nbextensions_configurator/tree_tab/main  enabled 
      - Validating: problems found:
        - require?  X nbextensions_configurator/tree_tab/main
  config dir: /home/dominique/.local/miniconda/envs/test/etc/jupyter/nbconfig
    notebook section
      jupyter-matplotlib/extension  enabled 
      - Validating: OK
      jupyter-js-widgets/extension  enabled 
      - Validating: OK
JupyterLab v3.2.2
/home/dominique/.local/miniconda/envs/test/share/jupyter/labextensions
        jupyter-matplotlib v0.10.2 enabled OK
        @jupyter-widgets/jupyterlab-manager v3.0.1 enabled OK (python, jupyterlab_widgets)

Installed from conda-forge

mamba create -n test matplotlib pandas seaborn jupyterlab ipympl -y -c conda-forge

More versions:

# Name                    Version                   Build  Channel
matplotlib                3.4.3            py39hf3d152e_1    conda-forge
matplotlib-base           3.4.3            py39h2fa2bec_1    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
pandas                    1.3.4            py39hde0f152_1    conda-forge
seaborn                   0.11.2               hd8ed1ab_0    conda-forge
seaborn-base              0.11.2             pyhd8ed1ab_0    conda-forge

Do you have an idea why this is happening?

ianhi commented 3 years ago

This worked in ipympl 0.8.0 but not in 0.8.2 so this may be something to do with the new display logic. Although interestingly just calling plt.gca (which is what seaborn does) seems to work.

ianhi commented 3 years ago

@mwaskom do you have any insight? it seems like you just call plt.gca() but I suppose something that's happening in plotter.plot is making things go poorly.

mwaskom commented 3 years ago

I think I'd want to know a few things

Is this a "seaborn" problem or a "seaborn.heatmap" problem?
If the latter, does a reasonably similar plot made only in matplotlib (i.e., pcolormesh with some texts on it) work?
If you can make a similar matplotlib plot with no issues, can you get the heatmap to work if you boil it down to the simplest version of what seaborn can draw? (i.e. no annotations, no colorbar, and disable the automatic selection of tick label density)

dominiquesydow commented 3 years ago

Thanks a lot for getting back to me so quickly! Let me try to apply your suggestions.

it seems like you just call plt.gca() but I suppose something that's happening in plotter.plot is making things go poorly.

Do you mean like this?

%matplotlib widget
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

df = pd.DataFrame([[10, 20, 30, 40], [50, 30, 8, 15], 
                   [25, 14, 41, 8], [7, 14, 21, 28]])
ax = plt.gca()
sns.heatmap(df, ax=ax)

If the latter, does a reasonably similar plot made only in matplotlib (i.e., pcolormesh with some texts on it) work?

Yes, it seems that works.

%matplotlib widget
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

df = pd.DataFrame([[10, 20, 30, 40], [50, 30, 8, 15], 
                   [25, 14, 41, 8], [7, 14, 21, 28]])

plt.pcolormesh(df)

for y in range(df.shape[0]):
    for x in range(df.shape[1]):
        plt.text(
            x + 0.5, y + 0.5, "%.1f" % df.iloc[x, y],
            horizontalalignment="center",
            verticalalignment="center",
        )

If you can make a similar matplotlib plot with no issues, can you get the heatmap to work if you boil it down to the simplest version of what seaborn can draw? (i.e. no annotations, no colorbar, and disable the automatic selection of tick label density)

%matplotlib widget
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns

df = pd.DataFrame([[10, 20, 30, 40], [50, 30, 8, 15], 
                   [25, 14, 41, 8], [7, 14, 21, 28]])
sns.heatmap(df)

The same behaviour with

sns.heatmap(df, xticklabels=False, yticklabels=False)

and

sns.heatmap(df, xticklabels=True, yticklabels=True)

assuming this is what was meant with

disable the automatic selection of tick label density

mwaskom commented 3 years ago

Yes that's right about the tick labels. But you could also try isolating the colorbar: it's on by default in the seaborn plot, so you could try turning it off there (cbar=False and turning it on for the matplotlib plot (plt.colorbar).

But I think that doesn't answer the other question which is ... can you use other seaborn functions, or is this an issue specific to heatmap?

ianhi commented 3 years ago

A crucial thing in these tests is probably to always start with plt.close('all') so that the behavior of plt.gca is consistent

ianhi commented 3 years ago

Also @dominiquesydow can you please post the code snippets as text so that they are copy-pasteable - thanks!

dominiquesydow commented 3 years ago

Hi @ianhi,

Also @dominiquesydow can you please post the code snippets as text so that they are copy-pasteable - thanks!

I have updated my comment with copy-pasteable code.

Also, I have tried to investigate your initial question (my apologies for not getting to at first)

Is this a "seaborn" problem or a "seaborn.heatmap" problem?

I set up this notebook; below a summary:

Other seaborn plots seem fine to me.
I tried two versions that show the same behaviour
- (1) Define figure and axes (fig, ax = plt.subplots(nrows=1, ncols=1)) and pass axis to seaborn functions (ax parameter)
- (2) Close all axes and then get current axis (plt.close('all') followed by ax = plt.gca())
Interesting observation: Maybe the seaborn heatmap tries to draw to the axis before the current axis (see the end of the posted notebook)
- Procedure: 1. I draw e.g. a scatter plot, 2. I draw heatmap without clearing axes
- Result: The heatmap is drawn on top of the previous plot.
Note: I see the same behaviour with Jupyter Lab and Jupyter Notebook
Note: I do not see any plots when running all cells at once; I have to run cell-by-cell

mwaskom commented 3 years ago

That behavior all looks correct to me. If you don't provide seaborn with an Axes target, it plots on the "current" Axes (creating it if necessary). The Jupyter inline backend calls plt.close behind the scenes after it executes every cell, but ipyml doesn't (so e.g. you can create a plot in one cell then update it by doing something in another one).

01baftb commented 3 years ago

dominiquesydow commented 3 years ago

Hi @mwaskom,

Thanks for getting back to me. I am showing a minimal example here to show what is still confusing me.

# First cell
%matplotlib widget
import matplotlib.pyplot as plt
import seaborn as sns
tips = sns.load_dataset("tips")

fig, ax = plt.subplots(nrows=1, ncols=1)
sns.scatterplot(data=tips, x="total_bill", y="tip", hue="time", ax=ax)

This works nicely, I get the plot:

Then, I call plt.close as you suggested:

# Second cell
plt.close()

Then, I try to plot sns.heatmap and get an empty plot:

# Third cell
import numpy as np

uniform_data = np.random.rand(10, 12)

fig, ax = plt.subplots(nrows=1, ncols=1)
sns.heatmap(uniform_data, ax=ax)

Do you have an idea on how to make the last plot appear?

If you don't provide seaborn with an Axes target, it plots on the "current" Axes (creating it if necessary). The Jupyter inline backend calls plt.close behind the scenes after it executes every cell, but ipyml doesn't (so e.g. you can create a plot in one cell then update it by doing something in another one).

With ax=ax, I am providing sns.heatmap with an Axes target (at least I think I am); I was expecting ipympl to use that Axes as well.

mwaskom commented 3 years ago

I'm confused about how what you're describing there is not represented in the notebook you shared? You seem to be plotting heatmap just fine there?

dominiquesydow commented 3 years ago

My apologies, apparently uploading the notebook to GH makes the heatmap appear... running the same notebook locally shows the behaviour discussed in this comment.

For the time being, I will pin ipympl to 0.8.0; here the seaborn heatmaps still appear as expected.

mwaskom commented 3 years ago

Oh that's interesting! I'd say this sounds like an issue on the ipympl side of things. I wish I could offer a hypothesis based on the fact that it seems to happen with heatmaps, but nothing comes immediately to mind beyond what I suggested above.

dominiquesydow commented 2 years ago

Thanks for taking the time to discuss this issue with me.

Although I cannot offer a solution with the latest ipympl version, downgrading to 0.8.0 helped; I am therefore closing this issue. Please re-open if more discussion is needed for a solution for the latest version.

ianhi commented 2 years ago

Please re-open if more discussion is needed for a solution for the latest version.

Yes this is definitely a bug than needs fixing - thanks for finding it!

matplotlib / ipympl

No heatmap shown with `seaborn` and `%matplotlib widget` #391

Describe the issue

Versions