search

pcdshub / BEAMS

Beamline Engineering Automation Management Systems
Other
1 stars 1 forks source link

Non determinsistic segault seemingly introduced by #41 #42

Open joshc-slac opened 6 days ago

joshc-slac commented 6 days ago

Current Behavior

The "old" behavior of a weird interaction of caproto and pytest has resurfaced since PR #41 . In the logs I am seeing a non deterministic segfault. This may be nothing, an edge case exercised by caproto and pytest that will never get us in the real world, it may also be real and I want to track it here..

joshc-slac commented 6 days ago

beams/tests/test_tree_generator.py ..Fatal Python error: Segmentation fault

Thread 0x00007fcadb1ff6c0 (most recent call first):
  File /usr/lib/python3.11/threading.py, line 320 in wait
  File /usr/lib/python3.11/multiprocessing/queues.py, line 231 in _feed
  File /usr/lib/python3.11/threading.py, line 975 in run
  File /usr/lib/python3.11/threading.py, line 1038 in _bootstrap_inner
  File /usr/lib/python3.11/threading.py, line 995 in _bootstrap

Current thread 0x00007fcae098c2c0 (most recent call first):
  Garbage-collecting
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 1896 in clear_subscription
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 670 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 546 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 353 in _clear_auto_monitor_subscription
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 1152 in disconnect
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 53 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 1165 in __del__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 1021 in connect_channel
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/ca.py, line 576 in wrapper
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 445 in connect
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 430 in wait_for_connection
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 46 in wrapped
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/pv.py, line 156 in get_pv
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/__init__.py, line 150 in caget
  File /home/josh/Code/playpen/BEAMS/beams/tree_config.py, line 148 in cond_func
  File /home/josh/Code/playpen/BEAMS/beams/behavior_tree/ActionNode.py, line 40 in work_wrapper
  File /usr/lib/python3.11/multiprocessing/process.py, line 108 in run
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/epics/multiproc.py, line 39 in run
  File /usr/lib/python3.11/multiprocessing/process.py, line 314 in _bootstrap
  File /usr/lib/python3.11/multiprocessing/popen_fork.py, line 71 in _launch
  File /usr/lib/python3.11/multiprocessing/popen_fork.py, line 19 in __init__
  File /usr/lib/python3.11/multiprocessing/context.py, line 281 in _Popen
  File /usr/lib/python3.11/multiprocessing/context.py, line 224 in _Popen
  File /usr/lib/python3.11/multiprocessing/process.py, line 121 in start
  File /home/josh/Code/playpen/BEAMS/beams/sequencer/helpers/Worker.py, line 42 in start_work
  File /home/josh/Code/playpen/BEAMS/beams/behavior_tree/ActionNode.py, line 99 in setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 110 in visited_setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 119 in setup
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/py_trees/trees.py, line 378 in setup
  File /home/josh/Code/playpen/BEAMS/beams/tests/test_tree_generator.py, line 53 in test_father_tree_execution
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/python.py, line 194 in pytest_pyfunc_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/python.py, line 1792 in runtest
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 169 in pytest_runtest_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 262 in <lambda>
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 341 in from_call
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 261 in call_runtest_hook
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 222 in call_and_report
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 133 in runtestprotocol
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/runner.py, line 114 in pytest_runtest_protocol
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 350 in pytest_runtestloop
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 325 in _main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 271 in wrap_session
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/main.py, line 318 in pytest_cmdline_main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_callers.py, line 102 in _multicall
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_manager.py, line 119 in _hookexec
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/pluggy/_hooks.py, line 501 in __call__
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/config/__init__.py, line 169 in main
  File /home/josh/Code/playpen/BEAMS/venv/lib/python3.11/site-packages/_pytest/config/__init__.py, line 192 in console_main
  File /home/josh/Code/playpen/BEAMS/venv/bin/pytest, line 8 in <module>

Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, google._upb._message, grpc._cython.cygrpc, apischema.serialization.methods, apischema.deserialization.methods (total: 18)
joshc-slac commented 5 days ago

fascinatingly removing this line: https://github.com/pcdshub/BEAMS/blob/b74af9a8b618147210afb75d1a292d7e6528cff7/beams/tests/conftest.py#L16 makes this error no longer occur. But then I don't get logging in my tests

tangkong commented 5 days ago

This makes sense, if the logging level is INFO (a higher severity than DEBUG), the logging record will get dropped before it gets processed. This does seem to implicate logging as the process that's getting left behind.

We probably need to figure out a good way of killing the logging thread after each test. Or at least figure out what pytest is doing with the logging messages