search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.42k stars 17.84k forks source link

BUG: MultiIndex label lookup with empty list indexers inconsistent wrt placement of indexer #54009

Open wence- opened 1 year ago

wence- commented 1 year ago

Pandas version checks

Reproducible Example

import pandas as pd
df = pd.DataFrame({"a": [1, 1, 2, 3], "b": [1, 0, 2, 2], "d": [1, 0, 2, 2], "c": [1, 2, 3, 4]}).set_index(["a", "b", "d"])

df.loc[([], [3]), :] # empty data frame
df.loc[([4], []), :] # keyerror

Issue Description

If missing scalar or list-like keys appear to the right of an empty list-like in an indexing tuple to a multi-index lookup, no KeyError is raised (and instead an empty dataframe is returned). In contrast, if the missing key is to the left of the empty list, then we do obtain a KeyError.

This appears to be a consequence of looking up from left to right with early exit for empty list likes.

Expected Behavior

It would be nice if the lookup consistently either raised a KeyError, or returned an empty dataframe.

An empty dataframe would be perhaps more performant (if one collected the indexers before applying them), but a KeyError seems to go by the principle of least surprise.

cc @attack68 who I think has had some opinions about whether multiindexing should raise keyerrors as often as normal indexing in the past.

Installed Versions

``` INSTALLED VERSIONS ------------------ commit : 6edff75126ba1500052b569e03a4567b98c8b8c7 python : 3.10.11.final.0 python-bits : 64 OS : Linux OS-release : 5.19.0-46-generic Version : #47~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Wed Jun 21 15:35:31 UTC 2 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_GB.UTF-8 LOCALE : en_GB.UTF-8 pandas : 2.1.0.dev0+1129.g6edff75126 numpy : 1.24.3 pytz : 2023.3 dateutil : 2.8.2 setuptools : 67.7.2 pip : 23.1.2 Cython : 0.29.33 pytest : 7.3.2 hypothesis : 6.79.1 sphinx : 6.2.1 blosc : None feather : None xlsxwriter : 3.1.2 lxml.etree : 4.9.2 html5lib : 1.1 pymysql : 1.0.3 psycopg2 : 2.9.3 jinja2 : 3.1.2 IPython : 8.14.0 pandas_datareader: None bs4 : 4.12.2 bottleneck : 1.3.7 brotli : fastparquet : 2023.4.0 fsspec : 2023.6.0 gcsfs : 2023.6.0 matplotlib : 3.7.1 numba : 0.57.0 numexpr : 2.8.4 odfpy : None openpyxl : 3.1.2 pandas_gbq : None pyarrow : 12.0.0 pyreadstat : 1.2.2 pyxlsb : 1.0.10 s3fs : 2023.6.0 scipy : 1.10.1 snappy : sqlalchemy : 2.0.16 tables : 3.8.0 tabulate : 0.9.0 xarray : 2023.5.0 xlrd : 2.0.1 zstandard : 0.19.0 tzdata : 2023.3 qtpy : None pyqt5 : None ```
wence- commented 1 year ago

The same kind of thing occurs for slices. If the sliced dataframe doesn't intersect with the pieces taken out by the other indexers (if they are not slices), then a keyerror is raised if the remaining indexers appear to the right of the slice, but not if they appear to the left.