attach_context failing on Mac OS X

[RobbieClarken]

I'm trying to use PyEPICS with multiple threads on Mac OS X 10.9.5 with EPICS base 3.14.12.5. All approaches seem to lead to channel access context errors. For example, the following code:

import epics
from epics.ca import CAThread

pv = epics.PV('SR11BCM01:CURRENT_MONITOR')

def callback():
    print(pv.get())

CAThread(target=callback).start()

will generate this traceback:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 1667, in run
    Thread.run(self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "app.py", line 7, in callback
    print(pv.get())
  File "/usr/local/lib/python2.7/site-packages/epics/pv.py", line 275, in get
    if not self.wait_for_connection():
  File "/usr/local/lib/python2.7/site-packages/epics/pv.py", line 219, in wait_for_connection
    ca.attach_context(self.context)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 347, in wrapper
    return fcn(*args, **kwds)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 444, in wrapper
    return PySEVCHK( fcn.__name__, status)
  File "/usr/local/lib/python2.7/site-packages/epics/ca.py", line 432, in PySEVCHK
    raise CASeverityException(func_name, message(status))
CASeverityException:  attach_context returned 'Thread is already attached to a client context'

The same code works fine on Linux. I have tried the other approaches from the "How to work with CA and Threads" documentation such as the withInitialContext decorator or calling epics.ca.use_initial_context() and these result in the same issue or cause a segmentation fault.

Am I doing something wrong or is threading broken for pyepics on OS X?

[klauer]

What pyepics version are you using? I submitted a quick fix for 3.2.4 that (I think) took care of this issue. That fix is here.

[RobbieClarken]

Unfortunately this problem is happening with the latest pyepics (3.2.4). I tried removing your check and without it Python crashes with a horrible segmentation fault so it is at least an improvement. Are you running Mavericks or Yosemite?

[klauer]

The following should work:

import epics
from epics.ca import CAThread

def callback():
    pv = epics.PV('SR11BCM01:CURRENT_MONITOR')
    print(pv.get())

CAThread(target=callback).start()

The key difference is that the PV instance is instantiated in the CAThread versus (in your case) the main thread. The PV instance then tries to switch CA contexts to that of the thread, which is what is failing. @newville, any ideas?

[newville]

@RobbieClarken @klauer Thanks, and I'm not sure what the real problem is. I'm just getting back from travel and trying to catch up on many things. I got as far as confirming the problem.

I also agree that it's probably most normal (and maybe even a good idea) to use PVs created within a particular thread, and not except PVs to be global. But: it's weird that this works on Linux (and Windows!) but not OSX. Investigating, but not sure how much time it will take to resolve...

[newville]

@RobbieClarken @klauer After some experimentation, I found that simply removing the @withSEVCHK decorator for attach_context() (that is, commenting out https://github.com/pyepics/pyepics/blob/master/lib/ca.py#L641) seems to avoid the problem. This change is now the "darwin_threading" branch (https://github.com/pyepics/pyepics/tree/darwin_threading).

Can you confirm that this works for you all in your example and "real code"? Of course, removing such checks should be done reluctantly, but It's not clear that the error from the severity check is actually correct here (is the message really just a warning?). Either way, I would want to test that on other platforms before pushing such a change, and may not get to a full run of the test suite until later in the week, or even next week.

[RobbieClarken]

Thanks for investigating this. In the example code the darwin_threading branch pyepics generates a new exception:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/ca.py", line 1670, in run
    Thread.run(self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "app.py", line 8, in callback
    print(pv.get())
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/pv.py", line 285, in get
    if ca.get_cache(self.pvname)['value'] is not None:
TypeError: 'NoneType' object has no attribute '__getitem__

If I change the test code to:

import epics
from epics.ca import CAThread

epics.caget('SR11BCM01:CURRENT_MONITOR')

def callback():
    print(epics.caget('SR11BCM01:CURRENT_MONITOR'))

CAThread(target=callback).start()

I get:

Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/ca.py", line 1670, in run
    Thread.run(self)
  File "/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "app.py", line 7, in callback
    print(epics.caget('SR11BCM01:CURRENT_MONITOR'))
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/__init__.py", line 88, in caget
    as_numpy=as_numpy)
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/pv.py", line 285, in get
    if ca.get_cache(self.pvname)['value'] is not None:
  File "/Users/robbie/Developer/ASLS/Robot/threadingtest/.venv/lib/python2.7/site-packages/epics/ca.py", line 278, in get_cache
    return _cache[current_context()].get(pvname, None)
KeyError: 140218349913280

I think there is a deeper problem with trying to use a new context for the thread. It would be nice to get to the bottom of why use_initial_context() is causing seg faults. I tried using the withInitialContext decorator on a different Mac running Yosemite and it also seg faults there.

[newville]

@RobbieClarken Well, the "deeper problem" may be a real difference with OSX (or llvm) vs what "normal Epics CA threads" expect. That is, I'm not certain that this is solely a problem with pyepics, or that we're going to solve it here. Suggestions welcome.

The error you're now seeing is mostly due to using the simple "epics.caget" interface (which wasn't used in your original version). If you use the PV interface (which the docs recommend, especially for threading), you shouldn't have any trouble. This

import time
import epics
from epics.ca import CAThread

pv = epics.PV('Py:ao1')
print(pv, pv.get())  # that is, make sure PV is actually connected.

def process():
    n = 0
    while n < 10:
        print(n, pv.get())
        time.sleep(0.25)
        n += 1

worker = CAThread(target=process)
worker.start()
worker.join()

works for me (after removing the severity check that is). And, again, in 'real code' that was multi-threaded you would probably not use caget() anyway, and you would probably not be referencing PVs created in the main thread from a worker thread. This is why I suggested testing 'real code'. We try to make simple things simple, but if you're looking to combine multi-threading and CA, you're beyond "simple", and sort of have to use the libraries as documented. Hope that helps.

[RobbieClarken]

Tested and everything seems to be work. Great work @newville and @klauer!