Processes killed and semaphore objects leaked when reading pandas data

arrow-py / arrow

🏹 Better dates & times for Python

https://arrow.readthedocs.io

Apache License 2.0

8.7k stars 673 forks source link

Processes killed and semaphore objects leaked when reading pandas data #1000

Closed Alalalalaki closed 3 years ago

Alalalalaki commented 3 years ago

Issue Description

When I run pa.Table.from_pandas(df) for a >1G dataframe, it reports

Killed: 9
anaconda3/envs/py38/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown

System Info

🖥 OS name and version: macOS 11.4
🐍 Python version: 3.8.10
🏹 Arrow version: 4.0.1

jadchaar commented 3 years ago

Hi @Alalalalaki, I don’t think this is related to arrow but is instead a Pandas issue. What made you think it was an arrow issue :)?

Alalalalaki commented 3 years ago

@jadchaar Sorry, I find I get the wrong place. I am confused pyarrow with here.