search

bluesky / ophyd

hardware abstraction in Python with an emphasis on EPICS
https://blueskyproject.io/ophyd
BSD 3-Clause "New" or "Revised" License
50 stars 79 forks source link

Making Ophyd Seg fault! #121

Closed ericdill closed 8 years ago

ericdill commented 9 years ago

So this is a fun one...

I'm trying to use the test cam 1 sim detector on xf23id1-ioc3. Instantiating XF:23ID1-ES{Tst-Cam:1} as a SimDetector and calling its read() function causes a seg fault. Any ideas @klauer @dchabot (long shot, @tacaswell) ?

(ophyd)edill@xf23id1-ioc3:~$ ipython
Python 2.7.9 (default, Jan  1 2015, 17:18:30) 
Type "copyright", "credits" or "license" for more information.

IPython 3.1.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from ophyd.controls.areadetector.detectors import SimDetector

In [2]: det1 = SimDetector('XF:23ID1-ES{Tst-Cam:1}')
/home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/pytz/__init__.py:29: UserWarning: Module bson was already imported from /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/bson/__init__.pyc, but /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/mongoengine-0.8.7-py2.7.egg is being added to sys.path
  from pkg_resources import resource_stream
2015-05-03 11:13:57,712 [ophyd_session:WARNING] Instantiating SessionManager outside of IPython
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 49925,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)

In [3]: det1.read()
2015-05-03 11:13:59,583 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:ImageMode_RBV connected
2015-05-03 11:13:59,585 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:ImageMode connected
2015-05-03 11:13:59,609 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:Acquire_RBV connected
2015-05-03 11:13:59,610 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:Acquire connected
2015-05-03 11:13:59,662 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:DetectorState_RBV connected
2015-05-03 11:13:59,934 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize1_RBV connected
2015-05-03 11:13:59,935 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize0_RBV connected
2015-05-03 11:13:59,936 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize2_RBV connected
2015-05-03 11:13:59,961 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:NDimensions_RBV connected
2015-05-03 11:13:59,987 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArrayData connected
Segmentation fault
stuwilkins commented 9 years ago

Eric:

2015-05-03 11:13:59,987 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArrayData connected

Why is ophyd connecting to the actual array data? I would suspect there is something here as that is a big array and most likely to cause buffer underruns. Which detector class are you using? The current area detector implementation doesn’t use that anywhere.

Also, have you tried using pyepics to connect to that PV? I would suspect this is a pyepics issue.

S

From: Eric Dill notifications@github.com<mailto:notifications@github.com> Reply-To: NSLS-II/ophyd reply@reply.github.com<mailto:reply@reply.github.com> Date: Sunday, May 3, 2015 at 11:21 AM To: NSLS-II/ophyd ophyd@noreply.github.com<mailto:ophyd@noreply.github.com> Subject: [ophyd] Making Ophyd Seg fault! (#121)

2015-05-03 11:13:59,987 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArrayData connected

stuwilkins commented 9 years ago

Which branch are you using?

S

From: Eric Dill notifications@github.com<mailto:notifications@github.com> Reply-To: NSLS-II/ophyd reply@reply.github.com<mailto:reply@reply.github.com> Date: Sunday, May 3, 2015 at 11:21 AM To: NSLS-II/ophyd ophyd@noreply.github.com<mailto:ophyd@noreply.github.com> Subject: [ophyd] Making Ophyd Seg fault! (#121)

2015-05-03 11:13:59,987 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArrayData connected

dchabot commented 9 years ago

I suspect you’ve found a bug (not sure where in the stack), or a configuration problem.

Before you launch ipython, try:

export EPICS_CA_MAX_ARRAY_BYTES=10000000

and repeat your experiment.

FWIW, this process works for me - I successfully get a 640x480 np array.

— dc

On May 3, 2015, at 11:21 AM, Eric Dill notifications@github.com wrote:

So this is a fun one...

I'm trying to use the test cam 1 sim detector on xf23id1-ioc3. Instantiating XF:23ID1-ES{Tst-Cam:1} as a SimDetector and calling its read() function causes a seg fault. Any ideas @klauer @dchabot (long shot, @tacaswell) ?

(ophyd)edill@xf23id1-ioc3:~$ ipython Python 2.7.9 (default, Jan 1 2015, 17:18:30) Type "copyright", "credits" or "license" for more information.

IPython 3.1.0 -- An enhanced Interactive Python. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://binstar.org ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.

In [1]: from ophyd.controls.areadetector.detectors import SimDetector

In [2]: det1 = SimDetector('XF:23ID1-ES{Tst-Cam:1}') /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/pytz/init.py:29: UserWarning: Module bson was already imported from /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/bson/init.pyc, but /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/mongoengine-0.8.7-py2.7.egg is being added to sys.path from pkg_resources import resource_stream 2015-05-03 11:13:57,712 [ophyd_session:WARNING] Instantiating SessionManager outside of IPython cas warning: Configured TCP port was unavailable. cas warning: Using dynamically assigned TCP port 49925, cas warning: but now two or more servers share the same UDP port. cas warning: Depending on your IP kernel this server may not be cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)

In [3]: det1.read() 2015-05-03 11:13:59,583 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:ImageMode_RBV connected 2015-05-03 11:13:59,585 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:ImageMode connected 2015-05-03 11:13:59,609 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:Acquire_RBV connected 2015-05-03 11:13:59,610 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:Acquire connected 2015-05-03 11:13:59,662 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}cam1:DetectorState_RBV connected 2015-05-03 11:13:59,934 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize1_RBV connected 2015-05-03 11:13:59,935 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize0_RBV connected 2015-05-03 11:13:59,936 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArraySize2_RBV connected 2015-05-03 11:13:59,961 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:NDimensions_RBV connected 2015-05-03 11:13:59,987 [ophyd_session:DEBUG] connection notification: XF:23ID1-ES{Tst-Cam:1}image1:ArrayData connected Segmentation fault

— Reply to this email directly or view it on GitHub.

ericdill commented 9 years ago

@dchabot thanks for the suggestion, but it's still segfaulting :(

ericdill commented 9 years ago

@stuwilkins branch: master detector class: ophyd.controls.areadetector.detectors.SimDetector

ericdill commented 9 years ago

@dchabot The stack trace is different with and without an "ophyd" ipython profile...

With the ipython profile defined in /home/edill/.ipython/profile_ophyd-testbed it segfaults immediately. However without the ipython profile (the original post shows this), it segfaults after connecting to a few PVs...

with the ipython profile:

(ophyd)edill@xf23id1-srv1:~$ ipython --profile=ophyd-testbed
Activating auto-logging. Current session state plus future input saved.
Filename       : /home/edill/.ophyd/logs/testbed_log.py
Mode           : append
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active
Python 2.7.9 (default, Jan  1 2015, 17:18:30) 
Type "copyright", "credits" or "license" for more information.

IPython 3.1.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

IPython profile: ophyd-testbed
Using matplotlib backend: Qt4Agg
Loading Ophyd Session Manager...
/home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/pytz/__init__.py:29: UserWarning: Module bson was already imported from /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/bson/__init__.pyc, but /home/edill/anaconda/envs/ophyd/lib/python2.7/site-packages/mongoengine-0.8.7-py2.7.egg is being added to sys.path
  from pkg_resources import resource_stream
cas warning: Configured TCP port was unavailable.
cas warning: Using dynamically assigned TCP port 49997,
cas warning: but now two or more servers share the same UDP port.
cas warning: Depending on your IP kernel this server may not be
cas warning: reachable with UDP unicast (a host's IP in EPICS_CA_ADDR_LIST)
2015-05-03 12:38:03,373 [ophyd_session:DEBUG] Last persistence list = ['_persisting', '_scan_id']
Stored '_persisting' (list)
2015-05-03 12:38:03,400 [ophyd_session:DEBUG] Last Scan ID = 117
Stored '_scan_id' (int)
...Done loading Ophyd Session Manager

In [1]: from ophyd.controls.areadetector.detectors import SimDetector

In [2]: det1 = SimDetector('XF:23ID1-ES{Tst-Cam:1}')

In [3]: det1.read()
Segmentation fault
stuwilkins commented 9 years ago

OK, what are you trying to achieve Eric? Why not use the AreaDetector class?

S On May 3, 2015, at 12:42 PM, Eric Dill notifications@github.com<mailto:notifications@github.com> wrote:

@stuwilkinshttps://github.com/stuwilkins branch: master detector class: ophyd.controls.areadetector.detectors.SimDetector

— Reply to this email directly or view it on GitHubhttps://github.com/NSLS-II/ophyd/issues/121#issuecomment-98502377.

ericdill commented 9 years ago

I'm trying to sort out the ophyd.controls.area_detector.py module and the ophyd.controls.areadetector package

klauer commented 9 years ago

What version of PyEpics is installed? There's an issue with segfaulting when grabbing timestamped large arrays, which should be fixed in 3.2.4.

stuwilkins commented 9 years ago

OK, how so?

S On May 3, 2015, at 1:02 PM, Eric Dill notifications@github.com<mailto:notifications@github.com> wrote:

I'm trying to sort out the ophyd.controls.area_detector.py module and the ophyd.controls.areadetector package

— Reply to this email directly or view it on GitHubhttps://github.com/NSLS-II/ophyd/issues/121#issuecomment-98504654.

stuwilkins commented 9 years ago

That would make a lot of sense.

S On May 3, 2015, at 1:02 PM, kne notifications@github.com<mailto:notifications@github.com> wrote:

What version of PyEpics is installed? There's an issue with segfaulting when grabbing timestamped large arrays, which should be fixed in 3.2.4.

— Reply to this email directly or view it on GitHubhttps://github.com/NSLS-II/ophyd/issues/121#issuecomment-98504659.

ericdill commented 9 years ago

PyEpics 3.2.4rc2

In [1]: import epics

In [2]: epics.__version__
Out[2]: '3.2.4rc2'
ericdill commented 9 years ago

The segfault is happening here

klauer commented 9 years ago

It should have been fixed by rc2, but please try the final 3.2.4 release or master of pyepics.

What's the size of the array, data type, and what's EPICS_CA_MAX_ARRAY_BYTES set to? If it still crashes with the above, set it to something huge like 100mb and re-run. If it doesn't crash then, it's still related to the issue I mentioned.

dchabot commented 9 years ago

It looks like there is a problem with the ioc itself. The ioc can produce images, but this is inconsistent - I periodically see detector-state==Aborted. I have obtained images via an opi screen as well as from an ophyd session.

The console indicates a possible hint at startup with:

recDynLinkOut: got CA context 2015/05/04 12:25:54.489 NDPluginCircularBuff::writeOctet error processing infix expression=, error=NULL or empty input argument to postfix() 2015/05/04 12:25:54.489 XF:23ID1-ES{Tst-Cam:1}CB1:TriggerCalc devAsynOctet::writeIt failed NDPluginCircularBuff:writeOctet: status=-1, function=60, value=

However, I’m not sure if this is even related...

If you’d like to continue with exploration, I have a well-exercised, simulated detector running from xf23id1-srv1 ("XF:31IDA-BI{Cam:Tbl}"). This may help until we can get the problem(s) fixed on “Tst-Cam:1”.

— dc

On May 3, 2015, at 2:01 PM, K Lauer notifications@github.com wrote:

It should have been fixed by rc2, but please try the final 3.2.4 release or master of pyepics.

What's the size of the array, data type, and what's EPICS_CA_MAX_ARRAY_BYTES set to? If it still crashes with the above, set it to something huge like 100mb and re-run. If it doesn't crash then, it's still related to the issue I mentioned.

— Reply to this email directly or view it on GitHub.

tacaswell commented 8 years ago

According to @klauer this has been fixed upstream by pyepics, please re-open if this is still an issue.