search

SeattleTestbed / geoip_server

Resolve IP addresses to geo coordinates (not including database)
MIT License
0 stars 1 forks source link

geoipserver hangs, logging output not useful #6

Open aaaaalbert opened 9 years ago

aaaaalbert commented 9 years ago

The geoipserver is prone to hangs, and the logging it provides isn't adequate to debug the problem.

From a recent hiccup that left the server hanging intermittently (until I ^C'd one of the threads and thus unblock it):

119.154.168.130 - - [07/Nov/2014 00:19:26] "POST / HTTP/1.0" 200 -
----------------------------------------
Exception happened during processing of request from ('119.154.168.130', 60918)
Traceback (most recent call last):
  File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
  File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
  File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
  File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 299, in handle_one_request
  File "/usr/lib/python2.5/socket.py", line 346, in readline
KeyboardInterrupt

Also,

blackbox.poly.edu - - [08/Dec/2014 02:50:06] "POST / HTTP/1.0" 200 -
----------------------------------------
Exception happened during processing of request from ('128.238.63.50', 58718)
Traceback (most recent call last):
  File "/usr/lib/python2.5/SocketServer.py", line 222, in handle_request
  File "/usr/lib/python2.5/SocketServer.py", line 241, in process_request
  File "/usr/lib/python2.5/SocketServer.py", line 254, in finish_request
  File "/usr/lib/python2.5/SocketServer.py", line 522, in __init__
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 316, in handle
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 310, in handle_one_request
  File "/usr/lib/python2.5/SimpleXMLRPCServer.py", line 477, in do_POST
  File "/usr/lib/python2.5/BaseHTTPServer.py", line 368, in send_response
  File "/usr/lib/python2.5/socket.py", line 262, in write
  File "/usr/lib/python2.5/socket.py", line 249, in flush
error: (32, 'Broken pipe')

Thanks @us341 for debugging support.

aaaaalbert commented 9 years ago

Seems like it suffices to connect to the server, e.g. with telnet or netcat, and leave the connection open, but not send anything. This allows further clients to connect still, but blocks off the actual logic (XML-RPC). Yikes.

Potential solutions:

aaaaalbert commented 8 years ago

The problem isn't too prevalent btw. The backup geoip server hung on a connection only last week, ~18 months after the first report of this issue. I've patched it using the "interim fix" from above, i.e. a five second socket timeout.