Subprocess client issue

[jacquelinegarrahan]

I'm working on a server that monitors some pvs, performs some model computation, and serves another set of pvs based on the computation results. This p4p server runs inside of a subprocess and I have found that the client tools are breaking, sometimes leading to network connection crashes if allowed to run too long. Also, I've seen that if the p4p imports happen after the subprocess has been started, this is no longer an issue.

I've put together a minimal example and have been able to replicate in a python 3..8 environment with the latest conda-forge p4p build (3.5.3). The pv to be monitored is served with softIocPVA -d demo.db, and is extremely minimal:

record(ai, "test:input1")
{
   field(PINI, "YES")
   field(VAL,  1)
   field(PREC, 3)
}

For testing, I'm performing puts to this pv using the p4p client tools:

$ python -m p4p.client.cli put test:input1=5

Below is the broken server:

import time
from threading import Event
import multiprocessing
from p4p.nt import NTScalar
from p4p.server import Server
from p4p.server.thread import SharedPV
from p4p.client.thread import Context

multiprocessing.set_start_method("fork")

def monitor_callback(V):
    print("Inside callback")

class PVServer(multiprocessing.Process):
    def __init__(self,exit_event, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.exit_event = exit_event

    def run(self):
        self.context = Context("pva")
        self.monitors = {"test:input1": self.context.monitor("test:input1", monitor_callback)}

        pv = SharedPV(nt=NTScalar('d'),
              initial=0.0)  

        @pv.put
        def handle(pv, op):
            pv.post(op.value()) 
            op.done()

        self.providers = [{
            'test:output1': pv, 
        }]

        server = Server(self.providers) 

        while not self.exit_event.is_set():
            time.sleep(1)

if __name__ == "__main__":
    exit_event = Event()
    server = PVServer(exit_event)

    try: 
        server.start()
        while True:
            time.sleep(1)

    except:
        server.exit_event.set()

Moving the p4p inputs into run fixes all client problems:

import time
from threading import Event
import multiprocessing

multiprocessing.set_start_method("fork")

def monitor_callback(V):
    print("Inside callback")

class PVServer(multiprocessing.Process):
    def __init__(self,exit_event, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.exit_event = exit_event

    def run(self):
        from p4p.nt import NTScalar
        from p4p.server import Server
        from p4p.server.thread import SharedPV
        from p4p.client.thread import Context

        self.context = Context("pva")
        self.monitors = {"test:input1": self.context.monitor("test:input1", monitor_callback)}

        pv = SharedPV(nt=NTScalar('d'), 
              initial=0.0) 

        @pv.put
        def handle(pv, op):
            pv.post(op.value()) 
            op.done()

        self.providers = [{
            'test:output1':pv, 
        }]

        server = Server(self.providers) 

        while not self.exit_event.is_set():
            time.sleep(1)

if __name__ == "__main__":
    exit_event = Event()
    server = PVServer(exit_event)

    try: 
        server.start()
        while True:
            time.sleep(1)

    except:
        server.exit_event.set()

Could this be a blocking issue with the context?

[mdavidsaver]

The root issue you are encountering is that the C/C++ libraries behind P4P can not safely be used in a child process after a fork(). One options you have found: import P4P in the child process. Another might be to use multiprocessing with the spawn or forkserver method. eg. from your first example, you might try:

...
import multiprocessing
multiprocessing.set_start_method('forkserver') # <<< insert
from p4p.nt import NTScalar
...

I haven't tested this, but by my reading of the documentation this should create the child "forkserver" process before P4P and the associated EPICS libraries are loaded.

[jacquelinegarrahan]

What particular libraries are of concern? The pyepics implementation of the fork-safe CAProcess clears the context and I'm wondering if there might be a similar approach here.

[jacquelinegarrahan]

On second thought, this is kind of silly and I'll be moving to spawn. Thanks for your time @mdavidsaver

[mdavidsaver]

I've started https://github.com/epics-base/epics-base/issues/211 to hopefully help google to help others to find this. CAProcess specifically has come up before and imo. it is an incomplete solution when combined with the fork method.