BUG: DF.plot.hist() fails with "ValueError: setting an array element with a sequence" when using numpy 1.24

[rbu]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

X = pd.DataFrame({
    "p": ["A", "B"],
    "x1": [1, 1],
    "x2": [np.nan, 100.0],
})

X.plot.hist(by="p")

Issue Description

Generating a histogram plot that contains NaN values raises a ValueError when using numpy 1.24.

Searching for the error message, #50342 popped up. I have confirmed that this is a different bug and that neither pandas 1.5.3 nor 2.0.0 which contain that commit fix this.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_76881/2078980972.py in <cell line: 7>()
      5 })
      6 
----> 7 X.plot.hist(by="p")

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_core.py in hist(self, by, bins, **kwargs)
   1349             >>> ax = df.plot.hist(column=["age"], by="gender", figsize=(10, 8))
   1350         """
-> 1351         return self(kind="hist", by=by, bins=bins, **kwargs)
   1352 
   1353     def kde(self, bw_method=None, ind=None, **kwargs) -> PlotAccessor:

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_core.py in __call__(self, *args, **kwargs)
    973                     data.columns = label_name
    974 
--> 975         return plot_backend.plot(data, kind=kind, **kwargs)
    976 
    977     __call__.__doc__ = __doc__

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/__init__.py in plot(data, kind, **kwargs)
     69             kwargs["ax"] = getattr(ax, "left_ax", ax)
     70     plot_obj = PLOT_CLASSES[kind](data, **kwargs)
---> 71     plot_obj.generate()
     72     plot_obj.draw()
     73     return plot_obj.result

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/core.py in generate(self)
    446         self._compute_plot_data()
    447         self._setup_subplots()
--> 448         self._make_plot()
    449         self._add_table()
    450         self._make_legend()

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/hist.py in _make_plot(self)
    163                 kwds["weights"] = weights
    164 
--> 165             y = reformat_hist_y_given_by(y, self.by)
    166 
    167             artists = self._plot(ax, y, column_num=i, stacking_id=stacking_id, **kwds)

~/mambaforge/envs/X/lib/python3.9/site-packages/pandas/plotting/_matplotlib/groupby.py in reformat_hist_y_given_by(y, by)
    136     """
    137     if by is not None and len(y.shape) > 1:
--> 138         return np.array([remove_na_arraylike(col) for col in y.T]).T
    139     return remove_na_arraylike(y)

ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

Expected Behavior

I would expect to see a plot, or at least an error message explaining what is wrong with the data.

With numpy 1.23.X and earlier, I was getting a plot and this warning originating from pandas code:

/home/mldev/mambaforge/envs/instinct/lib/python3.9/site-packages/pandas/plotting/_matplotlib/groupby.py:138: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array([remove_na_arraylike(col) for col in y.T]).T

Installed Versions

INSTALLED VERSIONS ------------------ commit : 478d340667831908b5b4bf09a2787a11a14560c9 python : 3.9.13.final.0 python-bits : 64 OS : Linux OS-release : 5.15.0-1035-azure Version : #42~20.04.1-Ubuntu SMP Wed Mar 1 19:17:41 UTC 2023 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : C.UTF-8 LANG : C.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.0.0 numpy : 1.24.2 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 59.5.0 pip : 23.0.1 Cython : None pytest : 7.2.2 hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 7.33.0 pandas_datareader: None bs4 : 4.12.0 bottleneck : None brotli : fastparquet : None fsspec : 2023.3.0 gcsfs : None matplotlib : 3.7.1 numba : None numexpr : 2.7.3 odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : 1.10.1 snappy : None sqlalchemy : 1.4.46 tables : 3.7.0 tabulate : 0.9.0 xarray : None xlrd : None zstandard : None tzdata : 2023.3 qtpy : 2.3.1 pyqt5 : None

[Julian048]

@rbu is there a reason you're using X.plot.hist instead of X.hist?

[Julian048]

To my understanding np.nan values should be allowed as they work when the 'by' parameter is not passed, but if anyone could elaborate whether or not they should be allowed when 'by' is passed I would appreciate that.

[Julian048]

In the Code it looks intentional that nan values are not accepted, so I think an error message should be added in that case to let people know that nan values are not possible with the parameter by != None. (Unless the functionality to accept nan values should be added)