BUG: Using tuples to specify single columns in .loc

[SudhanAnnamalai]

Pandas version checks

• [X] I have checked that this issue has not already been reported.

• [X] I have confirmed this bug exists on the latest version of pandas.

• [ ] I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(5,4), 
                  index = ["a", "b", "c", "d", "e"],
                  columns = ["One", "Two", "Three", (6, "two")])

Issue Description

Error line

df.loc[:, (6, "two")] #KeyError: "None of [Index([6, 'two'], dtype='object')] are in the [columns]" df.loc["a", (6, "two")] # Assertion Error : assert retval.ndim == self.ndim

Expected Behavior

However, the code below works fine(as expected)

1. df.loc["a", "Two"]
2. df.loc[:, "Two"]
3. df.loc[:, [(6, "two")]]
4. df.loc["a", [(6, "two")]]

Either, the syntax should work regardless of the datatype of the column_name, like in our example : tuple, or else the module should raise a exception or error with a clear information saying that the tuple based column name has to be changed.

I have also checked the above scenario, with other methods. Providing a single or multiple tuple based columne(s) results a data frame without error as in the above (3) (4) That could recommended along with the error statement, This could save a lot of time to the end user

Installed Versions

INSTALLED VERSIONS ------------------ commit : 2e218d10984e9919f0296931d92ea851c6a6faf5 python : 3.10.1.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.19045 machine : AMD64 processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : English_United States.1252 pandas : 1.5.3 numpy : 1.24.0 pytz : 2022.7.1 dateutil : 2.8.2 setuptools : 58.1.0 pip : 21.2.4 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : 4.9.2 html5lib : None pymysql : None psycopg2 : 2.9.6 jinja2 : 3.1.2 IPython : 8.12.0 pandas_datareader: None bs4 : 4.12.2 bottleneck : None brotli : None fastparquet : None fsspec : None gcsfs : None matplotlib : 3.6.2 numba : None numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : 1.10.0 snappy : None sqlalchemy : None tables : None tabulate : None xarray : None xlrd : None xlwt : None zstandard : None tzdata : 2022.7

[JostBrand]

Non single item nested Lists and Tuples are unpacked and matched individually, which is on purpose. Imagine the following example in which you want to specify which information to return. Both calls work but return different dataframes.

import pandas as pd
import numpy as np

df = pd.DataFrame(np.random.randn(5, 4), 
                  index = ["a", "b", "c", "d", "e"],
                  columns = [6, "two", "Three", (6, "two")])

df.loc[:, (6, "two")]
df.loc[:,[(6, "two")]]

Output 1:

	6	two
a	-1.84911	-1.58843
b	-1.258	0.433662
c	-0.401985	-0.654233
d	0.670789	0.92318
e	1.48344	0.683083

Output 2:

	(6, 'two')
a	1.09665
b	-0.709204
c	-0.174068
d	0.379871
e	0.0531941

[SudhanAnnamalai]

I totally agree with your point, but why would we unpack the tuple as columns while we can do that with the list based approach. Doesn't it create a confusion here?

Consider there is a big project which used .loc function to slice a part of data in which the column name is tuple based, and it throws error as I mentioned in the issue.

Do you think it's easy to debug the code with the error and exception raises.

Either the code should raise a appropriate exception or it should enforce a rule to restrict a user from using tuple based column names.

I am open to discuss a logical and valid points, and correct me if there is any other perspectives.

And I don't see a documentation which demands this squared bracket enclosed format while I want to fetch a single column, where the column name is a tuple.

[rhshadrach]

@SudhanAnnamalai - can you give this issue a descriptive title?

[SudhanAnnamalai]

@SudhanAnnamalai - can you give this issue a descriptive title?

Sure

[rhshadrach]

pandas does at times treat tuples and lists differently, and at first glance this seems to be a case where perhaps we should do so here. However it might be more complicated - for example, tuples are also used to indicate levels of a MultiIndex - so there is a number of cases to consider here.

cc @pandas-dev/pandas-core for any thoughts.

[attack68]

There are many cases to consider: see this thread (which may be a bit out of date in operation terms but not in function): https://github.com/pandas-dev/pandas/issues/39424#issuecomment-871670148

That specific link is a pathological example of multindexes where tuples form part of an element in a multiindex. Allowing tuples both as element labels and as containers for multinindex element labels in my opinion creates a plethora of problems without really giving a lot to users. Users could probably redefine their element tuples as something else.

@jbrockmendel proposed something else that didnt get much traction and didnt stir a discussion, unfortunately. https://github.com/pandas-dev/pandas/issues/42349