Broken pipe error after each request

[darix]

any idea what could cause that?

Aug 20 00:00:06 olddog galera-clustercheck[29704]: 172.17.42.2 - - [20/Aug/2015 00:00:06] "GET / HTTP/1.0" 200 -
Aug 20 00:00:06 olddog galera-clustercheck[29704]: Traceback (most recent call last):
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.process_request(request, client_address)
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.finish_request(request, client_address)
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.RequestHandlerClass(request, client_address, self)
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/SocketServer.py", line 657, in __init__
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.finish()
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/SocketServer.py", line 716, in finish
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.wfile.close()
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/socket.py", line 283, in close
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self.flush()
Aug 20 00:00:06 olddog galera-clustercheck[29704]: File "/usr/lib64/python2.7/socket.py", line 307, in flush
Aug 20 00:00:06 olddog galera-clustercheck[29704]: self._sock.sendall(view[write_offset:write_offset+buffer_size])
Aug 20 00:00:06 olddog galera-clustercheck[29704]: error: [Errno 32] Broken pipe

[darix]

ok this only happens when haproxy is connecting but not e.g. curl.

[darix]

aaaaaand there we go:

sendto(4, "HTTP/1.0 200 OK\r\n", 17, 0, NULL, 0) = 17
sendto(4, "Server: BaseHTTP/0.3 Python/2.7.10\r\n", 36, 0, NULL, 0) = -1 ECONNRESET (Connection reset by peer)
sendto(4, "Server: BaseHTTP/0.3 Python/2.7.10\r\n", 36, 0, NULL, 0) = -1 EPIPE (Broken pipe)

so i tried to move the whole request sending part into a function (reduces code duplication a bit too)

but i cant catch the IOError with it.

    def send_full_response(self, status_code, headers = { "Content-type": "text/html" }, body="Someone forgot a check response body"):
        try:
           self.send_response(status_code)
           for header, value in headers.items():
               self.send_header(header, value)
           self.end_headers()
           self.wfile.write(body)
        except IOError as e:
           # haproxy only reads the header ignore write errors here
           pass

....

self.send_full_response(status_code=503, body="Percona XtraDB Cluster Node state could not be retrieved.")

neither did it help to wrap

    try:
      httpd.serve_forever()
    except IOError as e:
      # haproxy only reads the header ignore write errors here
      #if e.errno == errno.EPIPE:
      pass

any ideas?

[Oneiroi]

@darix please take a look at the twisted implementation of the clustercheck on the branch here: https://github.com/Oneiroi/clustercheck/tree/twisted and the associated write up here, https://www.percona.com/blog/2014/05/16/benchmark-simplehttpserver-vs-pyclustercheck-twisted-implementation/ we've found the twisted implementation to not suffer from issues such as these previously, the only reason it's not merged into the master version atm is legacy support (twisted is not available on some older systems).

[darix]

JFYI: i also have a POC of a ruby implementation done now :D

[Oneiroi]

Closing issue