search

pandas-dev / pandas

Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more
https://pandas.pydata.org
BSD 3-Clause "New" or "Revised" License
42.57k stars 17.56k forks source link

BUG: random crash / hang when calculating rolling sum #59121

Closed e271p314 closed 3 days ago

e271p314 commented 3 days ago

Pandas version checks

Reproducible Example

import numpy as np
import pandas as pd

def func():
    cols = ['A', 'B', 'C']
    df = pd.DataFrame({c: np.random.randn(500000) for c in cols})
    sec_lst = [60 * m for m in range(1, 10)]
    sec_lst += [60 * m for m in range(10, 30, 5)]
    sec_lst += [60 * m for m in range(30, 60, 10)]
    sec_lst += [60 * m for m in range(60, 120, 15)]
    sec_lst += [60 * m for m in range(120, 180, 20)]
    sec_lst += [60 * m for m in range(180, 600, 30)]
    sec_lst += [3600 * h for h in range(10, 24)]
    df_sum_dict = {}
    for sec in sec_lst:
        for c in cols:
            try:
                df_sum_dict[f'{c}{sec}'] = df[c].rolling(sec).sum()
            except Exception as e:
                print(f"Error processing column {c} with window {sec}: {e}")
                continue

func()

Issue Description

running with this loop results in the following output (Terminated is the output when I kill it because the script hangs, should run few seconds at most)

$ for i in $(seq 1 100); do python test.py $i; if [ $? -ne 0 ]; then echo $i; fi; done
Segmentation fault (core dumped)
6
Terminated
24
Terminated
41
Segmentation fault (core dumped)
42
Segmentation fault (core dumped)
43
Terminated
44
Segmentation fault (core dumped)
45
Terminated
64
Terminated
65
Segmentation fault (core dumped)
74
Segmentation fault (core dumped)
95

Expected Behavior

it should not crash or hang

Installed Versions

INSTALLED VERSIONS ------------------ commit : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140 python : 3.12.3.final.0 python-bits : 64 OS : Linux OS-release : 6.8.0-35-generic Version : #35-Ubuntu SMP PREEMPT_DYNAMIC Mon May 20 15:51:52 UTC 2024 machine : x86_64 processor : x86_64 byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8 pandas : 2.2.2 numpy : 1.26.4 pytz : 2024.1 dateutil : 2.9.0.post0 setuptools : 70.0.0 pip : 24.0 Cython : None pytest : None hypothesis : None sphinx : None blosc : None feather : None xlsxwriter : None lxml.etree : None html5lib : None pymysql : None psycopg2 : None jinja2 : 3.1.4 IPython : 8.25.0 pandas_datareader : None adbc-driver-postgresql: None adbc-driver-sqlite : None bs4 : 4.12.3 bottleneck : 1.4.0 dataframe-api-compat : None fastparquet : None fsspec : 2024.6.0 gcsfs : 2024.6.0 matplotlib : 3.9.0 numba : 0.59.1 numexpr : None odfpy : None openpyxl : None pandas_gbq : None pyarrow : 16.1.0 pyreadstat : None python-calamine : None pyxlsb : None s3fs : None scipy : 1.13.1 sqlalchemy : 2.0.30 tables : None tabulate : None xarray : None xlrd : None zstandard : None tzdata : 2024.1 qtpy : None pyqt5 : None
chaoyihu commented 3 days ago

Hi, thanks for raising the issue! I was unable to reproduce the random crash/hang you experienced in any of the following envs:

Installed versions 1 (main) 

commit                : 5e972376fd5e4ab033f9922b546495d8efc9fda5
python                : 3.10.14.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 6.5.0-41-generic
Version               : #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2
machine               : x86_64
processor             : x86_64
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 3.0.0.dev0+1026.g5e972376fd
numpy                 : 1.26.4
pytz                  : 2024.1
dateutil              : 2.9.0.post0
setuptools            : 69.0.3
pip                   : 24.1
Cython                : 3.0.8
pytest                : 8.0.2
hypothesis            : 6.103.2
sphinx                : 7.3.7
blosc                 : None
feather               : None
xlsxwriter            : 3.1.9
lxml.etree            : 5.2.2
html5lib              : 1.1
pymysql               : 1.4.6
psycopg2              : 2.9.9
jinja2                : 3.1.2
IPython               : 8.12.3
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : 1.4.0
fastparquet           : 2024.5.0
fsspec                : 2024.6.0
gcsfs                 : 2024.6.0
matplotlib            : 3.9.0
numba                 : 0.59.1
numexpr               : 2.10.0
odfpy                 : None
openpyxl              : 3.1.4
pyarrow               : 15.0.2
pyreadstat            : 1.2.7
python-calamine       : None
pyxlsb                : 1.0.10
s3fs                  : 2024.6.0
scipy                 : 1.12.0
sqlalchemy            : 2.0.31
tables                : 3.9.2
tabulate              : 0.9.0
xarray                : 2024.6.0
xlrd                  : 2.0.1
zstandard             : 0.22.0
tzdata                : 2024.1
qtpy                  : 2.4.1
pyqt5                 : None
Installed versions 2 (close to the one provided in issue description) 

commit                : d9cdd2ee5a58015ef6f4d15c7226110c9aab8140
python                : 3.12.3.final.0
python-bits           : 64
OS                    : Linux
OS-release            : 6.5.0-41-generic
Version               : #41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 11:32:55 UTC 2
machine               : x86_64
processor             : x86_64
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 2.2.2
numpy                 : 1.26.4
pytz                  : 2024.1
dateutil              : 2.9.0
setuptools            : 70.1.1
pip                   : 24.0
Cython                : 3.0.10
pytest                : 8.2.2
hypothesis            : 6.104.1
sphinx                : 7.3.7
blosc                 : None
feather               : None
xlsxwriter            : 3.1.9
lxml.etree            : 5.2.2
html5lib              : 1.1
pymysql               : 1.4.6
psycopg2              : 2.9.9
jinja2                : 3.1.4
IPython               : 8.25.0
pandas_datareader     : None
adbc-driver-postgresql: None
adbc-driver-sqlite    : None
bs4                   : 4.12.3
bottleneck            : 1.4.0
dataframe-api-compat  : None
fastparquet           : 2024.5.0
fsspec                : 2024.6.0
gcsfs                 : 2024.6.0
matplotlib            : 3.8.4
numba                 : 0.60.0
numexpr               : 2.10.0
odfpy                 : None
openpyxl              : 3.1.4
pandas_gbq            : None
pyarrow               : 16.1.0
pyreadstat            : 1.2.7
python-calamine       : None
pyxlsb                : 1.0.10
s3fs                  : 2024.6.0
scipy                 : 1.14.0
sqlalchemy            : 2.0.31
tables                : 3.9.2
tabulate              : 0.9.0
xarray                : 2024.6.0
xlrd                  : 2.0.1
zstandard             : 0.22.0
tzdata                : 2024.1
qtpy                  : None
pyqt5                 : None

Since the reproducible script here is self-contained and works on my machine, I suspect this is not a problem with pandas or any of its dependencies. My intuition would be that the Segmentation fault (core dumped) errors here suggest low memory or even hardware issues - here is a potentially relevant post.

I would suggest double-checking memory usage / running the code on a different machine to see if the error persists.

e271p314 commented 3 days ago

Wow! you are right! sudo swapoff -a solved the problem for me, I would never never guess, thank you very much!