Server graceful shutdown concerns

[sergio-bershadsky]

Would be great to have some points in manual, describing ways to shutdown server without interrupting running processes and preventing new calls to start running.

[irmen]

You can do so by using a suitable loopCondition parameter https://pyro5.readthedocs.io/en/latest/servercode.html#running-the-request-loop (for an example on how to use this see the source module of the echoserver, where it gracefully shutdowns after a shutdown() call)

What additional information would you like to see in the manual?

[sergio-bershadsky]

Thanks! Looks like it will work for me. Will test it.

[sergio-bershadsky]

I need a gracefull shutdown on k8s environment. I will test it and return with some comments, thanks!

[sergio-bershadsky]

Tried to use the shutdown() method on daemon:

Here is how I start Pyro5 service:

def run(handler):

    daemon = Daemon(
        host=settings.get("host"),
        port=settings.get("port"),
        nathost=settings.get("public_host"),
        natport=settings.get("public_port"),
    )

    uri = daemon.register(publish(handler))

    ns = locate_ns()
    ns.register(
        f"handler/{handler.name}/{uuid.uuid4()}",
        uri=uri,
        metadata={
            "handler",
            f"name:{handler.name}",
            f"version:{handler.version}",
        }
    )

    # register graceful shutdown
    def exit_gracefully(signum, frame):
        print(f"Shutting down gracefully exit code: {signum}")
        daemon.shutdown()

    signal.signal(signal.SIGINT, exit_gracefully)
    signal.signal(signal.SIGTERM, exit_gracefully)

    print("Ready. Object uri =", uri)
    daemon.requestLoop()

The handler is smth like this

    @expose
    def run(self):
        for i in range(60):
            print(i)
            time.sleep(1)

Doing SIGINT leads to break run earlier than 60 seconds. The time between SIGINT and the actual shutdown is 5 seconds:

Pyro5/server.py:326

    def shutdown(self):
        """Cleanly terminate a daemon that is running in the requestloop."""
        log.debug("daemon shutting down")
        self.streaming_responses = {}
        time.sleep(0.02)
        self.__mustshutdown.set()
        if self.transportServer:
            self.transportServer.shutdown()
            time.sleep(0.02)
        self.close()
        self.__loopstopped.wait(timeout=5)  # use timeout to avoid deadlock situations

I am Expecting that actual shutdown appears just after the run returns its value

Als I am getting exceptions on client:

    return self.__send(self.__name, args, kwargs)
  File "/home/sergey/work/datazio/tabreport-cli2/venv/lib/python3.6/site-packages/Pyro5/client.py", line 221, in _pyroInvoke
    msg = protocol.recv_stub(self._pyroConnection, [protocol.MSG_RESULT])
  File "/home/sergey/work/datazio/tabreport-cli2/venv/lib/python3.6/site-packages/Pyro5/protocol.py", line 187, in recv_stub
    header = connection.recv(6)  # 'PYRO' + 2 bytes protocol version
  File "/home/sergey/work/datazio/tabreport-cli2/venv/lib/python3.6/site-packages/Pyro5/socketutil.py", line 435, in recv
    return receive_data(self.sock, size)
  File "/home/sergey/work/datazio/tabreport-cli2/venv/lib/python3.6/site-packages/Pyro5/socketutil.py", line 159, in receive_data
    raise err
Pyro5.errors.ConnectionClosedError: receiving: not enough data

[sergio-bershadsky]

The only solution I found for now, is to check worker thread pool before shutting down:

with daemon.transportServer.pool.busy

Works only with thread type transportServer

    # register graceful shutdown
    def exit_gracefully(signum, frame):
        daemon.transportServer.shutting_down = True
        print(f"Shutting down gracefully exit code: {signum} active threads: {len(daemon.transportServer.pool.busy)}")
        while len(daemon.transportServer.pool.busy):
            time.sleep(1)
        daemon.shutdown()

    signal.signal(signal.SIGINT, exit_gracefully)
    signal.signal(signal.SIGTERM, exit_gracefully)

[irmen]

You cannot gracefully abort running threads, so I suppose your solution/workaround is valid and needed in this case. Another solution may be to just exit() the whole server process at shutdown?

[irmen]

For more/better control of a server shutdown, you can also try the multiplex server instead of the thread based one. It won't have shutdown issues caused by the use of threads. Closing this issue now, feel free to reopen/add comments if desired