EOFError in Server Manager

[ahankinson]

Is there an existing issue for this?

• [X] I have searched the existing issues

Describe the bug

I have Sentry monitoring my Sanic server processes. Increasingly, I am seeing EOFError exceptions. A minimal stack trace is this:

EOFError: null
  File "sanic/mixins/startup.py", line 1146, in serve
    manager.run()
  File "sanic/worker/manager.py", line 189, in run
    self.monitor()
  File "sanic/worker/manager.py", line 294, in monitor
    self._sync_states()
  File "sanic/worker/manager.py", line 427, in _sync_states
    state = self.worker_state[process.name].get("state")
  File "<string>", line 2, in __getitem__
  File "multiprocessing/managers.py", line 822, in _callmethod
    kind, result = conn.recv()
  File "multiprocessing/connection.py", line 254, in recv
    buf = self._recv_bytes()
  File "multiprocessing/connection.py", line 418, in _recv_bytes
    buf = self._recv(4)
  File "multiprocessing/connection.py", line 387, in _recv
    raise EOFError

I am running Sanic with the following command (formatted as a python list):

[
/path/to/.venv/bin/sanic, 
module_name.server.app, 
--host, 
[my::IP::v6::address], 
--port, 
8003, 
--fast, 
--no-access-logs
]

This is run through a systemd unit file:

[Unit]
Description=My Description

[Service]
User=svc-user
Group=svc-user
Restart=always
RestartSec=5
TimeoutSec=5
WorkingDirectory=/path/to/application/
ExecStart=/path/to/application/.venv/bin/sanic module_name.server.app --host [my::IP::v6::address] --port 8003 --fast --no-access-logs

[Install]
WantedBy=multi-user.target

I'm attaching a PDF of the full stack trace that Sentry is reporting. I apologize that it's not in a more convenient form, but I couldn't figure out how to get it out in a readable manner. stack-trace.pdf

Code snippet

There does not seem to be any interaction with my own application code.

Expected Behavior

No response

How do you run Sanic?

Sanic CLI

Operating System

Linux

Sanic Version

23.12.1

Additional context

Ubuntu 22.04 (Linux-5.15.0-89-generic-x86_64-with-glibc2.35)

Python 3.11.0

[ahopkins]

Any way I can reproduce this?

[ahankinson]

Unfortunately I'm not getting any information about any requests that are triggering this, other than what is in the stack trace.

My current working hypothesis is that it's a client that prematurely closes the request. The only other clue is that sometimes it's failing with the buf variable being something like:

b'\x00\x00\x00*\x80\x04\x95\x1f\x00\x00\x00\x00\x00\x00\x00(\x8c\x0c7f7c1afd5f40\x94\x8c\x06values\x94)}\x94t\x94.'

I don't know how to decode it, nor do I know if it's useful if I did. It is never the same value, though.