[Python] Segmentation Fault via pytest-runner

[asfimport]

When running pytest on projects using pyarrow==0.14.0 on Linux, I am getting segmentation faults, but interestingly only when run via pytest-runner (which provides the setup.py pytest command)

This works (i.e. pytest directly):

$ pytest

Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.09s):
1 passed

However, this does not work, ending in a segmentation fault, even though the tests pass:

$ python setup.py pytest

running pytest
running egg_info
writing pyarrow_issue.egg-info/PKG-INFO
writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
writing requirements to pyarrow_issue.egg-info/requires.txt
writing top-level names to pyarrow_issue.egg-info/top_level.txt
reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
running build_ext

Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.07s):
1 passed
zsh: segmentation fault (core dumped) python setup.py pytest

backtrace from gdb

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0

(gdb) bt
#0 0x00007ffff7c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
#1 0x00007ffff7ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
#2 0x00007ffff023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
from /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
#3 0x00007fffed5e6467 in std::unordered_map<std::string, std::shared_ptr<arrow::ExtensionType>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<arrow::ExtensionType> > > >::~unordered_map() ()
from /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
#4 0x00007ffff7de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
#5 0x00007ffff7de5fae in exit () from /usr/lib/libc.so.6
#6 0x00007ffff7dcfeea in __libc_start_main () from /usr/lib/libc.so.6
#7 0x000055555555505e in _start ()

I have observed this behaviour on my machine running natively, and also via docker. Also, 0.13.0 does not exhibit this behaviour

 

Environment: $ uname -a Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux

$ python --version Python 3.7.3

$ pip freeze | grep -P "(pyarrow|pytest)" pyarrow==0.14.0 pytest==5.0.0 pytest-benchmark==3.2.2 pytest-cov==2.7.1 pytest-env==0.6.2 pytest-forked==1.0.2 pytest-html==1.21.1 pytest-metadata==1.8.0 pytest-mock==1.10.4 pytest-runner==5.1 pytest-sugar==0.9.2 pytest-xdist==1.29.0 Reporter: Josh Bode Assignee: Wes McKinney / @wesm

Original Issue Attachments:

• pyarrow-issue.tar.bz2
• pytest.log
• pytest-runner.log

  PRs and other links:

• GitHub Pull Request #4824

_{Note: This issue was originally created as ARROW-5863. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche: [~joshbode] one difference between pytest and pytest setup.py runner is that the second one is actually building pyarrow (there is a "running build_ext"), which is probably not what you wanted?

How did you install pyarrow? It is an installation from a wheel? Or did you install and build pyarrow from source yourself?

[asfimport]

Josh Bode: I installed it from the wheel on PyPI:

$ pip install pyarrow

Collecting pyarrow
Using cached https://files.pythonhosted.org/packages/8f/fa/407667d763c25c3d9977e1d19038df3b4a693f37789c4fe1fe5c74a6bc55/pyarrow-0.14.0-cp37-cp37m-manylinux2010_x86_64.whl
Requirement already satisfied: six>=1.0.0 in /home/josh/.virtualenvs/default/lib/python3.7/site-packages (from pyarrow) (1.12.0)
Requirement already satisfied: numpy>=1.14 in /home/josh/.virtualenvs/default/lib/python3.7/site-packages (from pyarrow) (1.16.4)
Installing collected packages: pyarrow
Successfully installed pyarrow-0.14.0

[asfimport]

Josh Bode: Also, it doesn't appear to be installing any package when I'm using python setup.py pytest since if I manually revert to 0.13.0 the issue goes away

[asfimport]

Wes McKinney / @wesm: At a glance, the problem appears to be coming from a race condition in interpreter teardown. @pitrou would know more

[asfimport]

Joris Van den Bossche / @jorisvandenbossche: I don't think pytest-runner is meant to run the tests of an installed wheel. When invoking python setup.py pytest it seems that it tries to build the package, but you have a wheel distribution, not a source (I am a bit surprised that the build_ext step doesn't error).

But since your system was not set up to build pyarrow (I don't think that the wheel itself is sufficient), it doesn't seem surprising that you get segfaults. But, you say that with 0.13.0, this does actually work? Can you show the log of that?

[asfimport]

Antoine Pitrou / @pitrou: I can take a look when I come back in ~10 days. Wes is right that it looks like a teardown-related issue.

[asfimport]

Josh Bode: Interesting - if I import pyarrow first, there is no segfault:

$ python -c "import pyarrow; import setup" pytest

running pytest
running egg_info
writing pyarrow_issue.egg-info/PKG-INFO
writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
writing requirements to pyarrow_issue.egg-info/requires.txt
writing top-level names to pyarrow_issue.egg-info/top_level.txt
reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
running build_ext
Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.02s):
1 passed

[asfimport]

Josh Bode: Here is the log with 0.13.0:

$ pip install pyarrow==0.13.0
Collecting pyarrow==0.13.0
Using cached https://files.pythonhosted.org/packages/3f/6c/91a3d949fe0763e60ac181b7b79e74e848e33e402e5e8274cad455519d76/pyarrow-0.13.0-cp37-cp37m-manylinux1_x86_64.whl
Requirement already satisfied: six>=1.0.0 in /home/josh/.virtualenvs/default/lib/python3.7/site-packages (from pyarrow==0.13.0) (1.12.0)
Requirement already satisfied: numpy>=1.14 in /home/josh/.virtualenvs/default/lib/python3.7/site-packages (from pyarrow==0.13.0) (1.16.4)
Installing collected packages: pyarrow
Found existing installation: pyarrow 0.14.0
Uninstalling pyarrow-0.14.0:
Successfully uninstalled pyarrow-0.14.0
Successfully installed pyarrow-0.13.0

$ python setup.py pytest
running pytest
running egg_info
writing pyarrow_issue.egg-info/PKG-INFO
writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
writing requirements to pyarrow_issue.egg-info/requires.txt
writing top-level names to pyarrow_issue.egg-info/top_level.txt
reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
running build_ext
Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.06s):
1 passed

[asfimport]

Antoine Pitrou / @pitrou: Not surprising. The code in the C backtrace didn't exist in 0.13.0 :-)

[asfimport]

Josh Bode: Also, pytest-runner is probably not to blame here, since calling the pytest distutils entry-point code directly works fine:

$ python -c "import setuptools, ptr; cmd = ptr.PyTest(setuptools.Distribution()); cmd.run_tests()"
Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.08s):
1 passed

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:

Also, pytest-runner is probably not to blame here, since calling the pytest distutils entry-point code directly works fine:

Note that what you show above does not seem to be building the C extensions of pyarrow, in contrast with the log you showed from pytest-runner

Further question, to have a reproducible workflow for the issue: from where are you running the tests? From within the site-packages where the wheel was installed? Although you seem to be running it from "/home/josh/scratch/pyarrow-issue"

[asfimport]

Josh Bode: I've attached an example (pyarrow-issue.tar.bz2) and I run the tests from the example directory (e.g. pyarrow-issue)

[asfimport]

Joris Van den Bossche / @jorisvandenbossche: Ah, sorry, I missed that archive with the example (only looked at the logs). So it is that example package that is being built, not pyarrow. So ignore my previous comments ;)

[asfimport]

Josh Bode: @jorisvandenbossche  - if I comment out the parts of setuptools that call build_ext (which is called unconditionally as part of the test entry-point that pytest-runner extends) I still get the segfault

[asfimport]

Josh Bode: What is interesting, though is that if I comment out line 172 from setuptools/command/test.py the segfault goes away:

sys.path[:] = old_path
#sys.modules.clear()
sys.modules.update(old_modules)
working_set.__init__()

which maybe isn't that surprising

[asfimport]

Josh Bode: Minimum working example :)

#! /usr/bin/env python

import pyarrow
import sys

del sys.modules['pyarrow.lib']

yields the segfault

[asfimport]

Wes McKinney / @wesm: Issue resolved by pull request 4824 https://github.com/apache/arrow/pull/4824