search

bendikro / supervisord-dependent-startup

An event listener for supervisord that does ordered startup
Apache License 2.0
57 stars 12 forks source link

BrokenPipe exception #10

Open ts-kst opened 3 years ago

ts-kst commented 3 years ago

Hi,

After having supervisord_dependent_startup waiting for hours for a process to reach the correct state, we saw following exception, it there something that can be done? Re-create the socket maybe?

Traceback (most recent call last): File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 910, in run main() File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 904, in main event_listener.run_and_listen() File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 829, in run_and_listen self.listen() File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 822, in listen for l in self._listen(): File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 817, in _listen event_parsed = self.handle_event(headers, payload) File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 757, in handle_event self.services_handler.update_proc_info() File "/usr/local/lib/python3.6/site-packages/supervisord_dependent_startup/supervisord_dependent_startup.py", line 472, in update_proc_info info = self.rpc.supervisor.getAllProcessInfo() File "/usr/lib64/python3.6/xmlrpc/client.py", line 1112, in call return self.send(self.name, args) File "/usr/lib64/python3.6/xmlrpc/client.py", line 1452, in request verbose=self.verbose File "/usr/local/lib/python3.6/site-packages/supervisor/xmlrpc.py", line 535, in request self.connection.request('POST', handler, request_body, self.headers) File "/usr/lib64/python3.6/http/client.py", line 1269, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1315, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1264, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.6/http/client.py", line 1040, in _send_output self.send(msg) File "/usr/lib64/python3.6/http/client.py", line 1000, in send self.sock.sendall(data) BrokenPipeError: [Errno 32] Broken pipe

bendikro commented 3 years ago

Hi

Haven't tested with processes that need hours to reach a state, so I never encountered this issue. Recreating self.rpc might be a solution, but I'd have to test to say for sure. I don't actually use this plugin at the moment so you'll have to modify the code and test this yourself I'm afraid.

ts-kst commented 3 years ago

You mean to do this: self.rpc = childutils.getRPCInterface(os.environ)

everywhere, when self.rpc.supervisor is used? when BrokenPipe ex. occur?

bendikro commented 3 years ago

You mean to do this: self.rpc = childutils.getRPCInterface(os.environ)

everywhere, when self.rpc.supervisor is used? when BrokenPipe ex. occur?

I presume that could be a solution, though I would try to avoid too much duplicate code and maybe handle this in a separate function is required. Without actually testing this myself I can't really recommend anything but trying and see how that goes :-)