Pagerank crash - Githubissues

seibert commented 3 years ago

There seems to be an intermittent segfault in the pagerank test. So far I can only reproduce it on Linux with:

pytest -v --forked -k test_pagerank

(--forked comes from the pytest-forked plugin, which allows pytest to run each test in a sub-process for better isolation.) I get no useful output besides:

==================================================================== test session starts ====================================================================
platform linux -- Python 3.8.10, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/sseibert/miniconda3/envs/mgng/bin/python
cachedir: .pytest_cache
rootdir: /home/sseibert/repo/mlir-graphblas, configfile: setup.cfg, testpaths: mlir_graphblas/tests
plugins: forked-1.3.0, cov-2.12.1
collected 178 items / 177 deselected / 1 selected

mlir_graphblas/tests/test_algorithms.py::test_pagerank Fatal Python error: Segmentation fault

Current thread 0x00007f0a3f9af740 (most recent call first):
  File "/home/sseibert/repo/mlir-graphblas/mlir_graphblas/engine.py", line 980 in python_callable
  File "/home/sseibert/repo/mlir-graphblas/mlir_graphblas/algorithms.py", line 470 in pagerank
  File "/home/sseibert/repo/mlir-graphblas/mlir_graphblas/tests/test_algorithms.py", line 160 in test_pagerank
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pytest_forked/__init__.py", line 62 in runforked
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 65 in _child
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/py/_process/forkedfunc.py", line 50 in __init__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pytest_forked/__init__.py", line 67 in forked_run_report
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pytest_forked/__init__.py", line 46 in pytest_runtest_protocol
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/main.py", line 323 in _main
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/home/sseibert/miniconda3/envs/mgng/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/sseibert/miniconda3/envs/mgng/bin/pytest", line 11 in <module>
FAILED                                                                                         [100%]

========================================================================= FAILURES ==========================================================================
_______________________________________________________________________ test_pagerank _______________________________________________________________________
:-1: running the test CRASHED with signal 11

seibert commented 3 years ago

I should note that if I leave out --forked, the test passes.

jim22k commented 3 years ago

Some bizarre findings. The segfault is happening in the 2nd call to mlalgo.pagerank. A single call is always fine, but the next call causes the segfault.

The problem goes away if any of the following are true:

Remove the line irb.util.del_sparse_tensor(prev_score)
Force recompilation of the algorithm each time it is run (by setting _pagerank = None prior to 2nd call)
Call scipy.sparse.csr_matrix() anytime prior to the compilation of pagerank
- This is why --forked is required to see the problem. Without that, test_left_bipartite_project_and_filter is always called first and makes a scipy csr_matrix, somehow affecting the way pagerank is compiled and avoiding this issue
Set a Python breakpoint prior to pr = _pagerank(graph, damping, tol, maxiter), then simply continue
Run on a Mac

Suffice it to say that I'm completely baffled. For now, I'm going to remove the del_sparse_tensor line.

metagraph-dev / mlir-graphblas

Pagerank crash #163