Hanging on recieve

[tlaundal]

I have a small daemon that repeatedly gets the status of a list of servers to find out how many players they have online. The deamon is single-threaded and consists mostly of a loop and calls to mcstatus' MinecraftServer.status.

After running for a couple of hours, the daemon suddenly hangs, and the daemon process uses over 90% of the system CPU. The function that does the server calls is decorated by a @timeout_decorator.timeout(30), to prevent the whole app from stopping, but this doesn't work.

This is all running on Ubuntu 14.04.1 LTS, with mcstatus==2.0.

Here is the stack trace from when i do Ctrl + C to kill the daemon:

Traceback (most recent call last):
  File "loader.py", line 22, in <module>
    main()
  File "loader.py", line 13, in main
    do_ping(app, db)
  File "/var/www/PlayerCountGraph/env/src/timeout-decorator/timeout_decorator/timeout_decorator.py", line 46, in new_f
    result = f(*args, **kwargs)
  File "/var/www/PlayerCountGraph/playercountgraph/scheduler.py", line 15, in do_ping
    ping_server(db, server)
  File "/var/www/PlayerCountGraph/playercountgraph/scheduler.py", line 29, in ping_server
    players = new_ping(server)
  File "/var/www/PlayerCountGraph/playercountgraph/scheduler.py", line 45, in new_ping
    .online ## This is really a call like this: MinecraftServer..status().players.online
  File "/var/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/server.py", line 57, in status
    result = pinger.read_status()
  File "/var/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/pinger.py", line 34, in read_status
    response = self.connection.read_buffer()
  File "/var/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 115, in read_buffer
    length = self.read_varint()
  File "/var/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 44, in read_varint
    part = ord(self.read(1))
  File "/var/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 143, in read
    result.extend(self.socket.recv(length - len(result)))
KeyboardInterrupt

From the stack trace, it seems the freezing/hanging happens in mcstatus. Is there any fix that could be applied to mcstatus, or a work around I could use?

[tlaundal]

Here I have another stack trace from when it hangs. Apparently, it does loop, but something is wrong with the receiving.

Here it had been hanging for 20 minutes.

Traceback (most recent call last):
  File "/srv/www/PlayerCountGraph/loader.py", line 25, in <module>
    main()
  File "/srv/www/PlayerCountGraph/loader.py", line 16, in main
    do_ping(app, db)
  File "/srv/www/PlayerCountGraph/env/src/timeout-decorator/timeout_decorator/timeout_decorator.py", line 46, in new_f
    result = f(*args, **kwargs)
  File "/srv/www/PlayerCountGraph/playercountgraph/scheduler.py", line 16, in do_ping
    ping_server(db, server)
  File "/srv/www/PlayerCountGraph/playercountgraph/scheduler.py", line 30, in ping_server
    players = new_ping(server)
  File "/srv/www/PlayerCountGraph/playercountgraph/scheduler.py", line 46, in new_ping
    .online
  File "/srv/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/server.py", line 57, in status
    result = pinger.read_status()
  File "/srv/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/pinger.py", line 34, in read_status
    response = self.connection.read_buffer()
  File "/srv/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 115, in read_buffer
    length = self.read_varint()
  File "/srv/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 44, in read_varint
    part = ord(self.read(1))
  File "/srv/www/PlayerCountGraph/env/lib/python3.4/site-packages/mcstatus/protocol/connection.py", line 142, in read
    while len(result) < length:
KeyboardInterrupt

[tlaundal]

I've added a couple of debug messages to mcstatus' source, so I can figure out what's going on with the loop. The print how many bytes that should be recieved(length) and after each iteration, how many have been received.

[tlaundal]

The debug messages paid off, and I now have some more debug information. The problem is that this loop never finishes. In both my cases where it crashed while printing the debug messages, the result bytearray have stayed at length 0, and in turn made the loop just run over and over again. This causes both the application using mcstatus to hang, and also eats up system resources.

I'll push a fix later today which just checks whether it actually received something, and if it did not, throws some error.