search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
43.83k stars 18k forks source link

BUG: errant behavior of ~ operator within an apply statement #60380

Closed ngeorgette closed 12 hours ago

ngeorgette commented 1 day ago

Pandas version checks

Reproducible Example

import pandas as pd

df = pd.DataFrame({"a": [1, 2, np.nan, 3]})

df = df.assign(not_null_assign = lambda x: (~pd.isna(x.a))) # correct

df['not_null_apply'] = df.apply(lambda x: True if (~pd.isna(x.a)) else False, axis=1) #incorrect

df['not_null_apply_and'] = df.apply(lambda x: True if (~pd.isna(x.a) & (1==1)) else False, axis=1) # correct

Issue Description

The use of the bitwise NOT ~ seems inconsistent in the above examples. I wonder if there is a bug in how pandas interprets apply when ~ is used within the conditional statement.

Expected Behavior

I would expect all three to correctly report whether the value is null.

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d07b4858144c219b9346329027024102ab6 python : 3.9.12.final.0 python-bits : 64 OS : Darwin OS-release : 23.4.0 Version : Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:06 PST 2024; root:xnu-10063.101.15~2/RELEASE_ARM64_T8103 machine : x86_64 processor : i386 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.4.2 numpy : 1.21.5 pytz : 2021.3 dateutil : 2.8.2 pip : 21.2.4 setuptools : 61.2.0 Cython : 0.29.28 pytest : 7.1.1 hypothesis : None sphinx : 4.4.0 blosc : None feather : None xlsxwriter : 3.0.3 lxml.etree : 4.8.0 html5lib : 1.1 pymysql : None psycopg2 : None jinja2 : 3.1.2 IPython : 8.3.0 pandas_datareader: None bs4 : 4.11.1 bottleneck : 1.3.4 brotli : fastparquet : None fsspec : 2022.02.0 gcsfs : None markupsafe : 2.0.1 matplotlib : 3.5.1 numba : 0.55.1 numexpr : 2.8.1 odfpy : None openpyxl : 3.0.9 pandas_gbq : None pyarrow : None pyreadstat : None pyxlsb : None s3fs : None scipy : 1.7.3 snappy : None sqlalchemy : 1.4.32 tables : 3.6.1 tabulate : None xarray : None xlrd : 2.0.1 xlwt : 1.3.0 zstandard : None

rhshadrach commented 12 hours ago

Thanks for the report, this is not due to pandas; it is just Python.

print(pd.isna(np.nan))
# True
print(~pd.isna(np.nan))
# -2
print(~True)
# -2
print(~True & (1==1))
# 0

Closing.