I think this will resolve the root cause of the error. My theory goes like this:
The server sends many messages to the client.
At some point, a full message doesn't flush across the socket.
The client doesn't have any means to know whether it's received a full message before attempting to deserialize it because we used NoFraming around messages.
The deserializer fails because the lazy bytestring chunk it received was an incomplete message.
The client gets a deserialization error and breaks the connection.
The server doesn't know the connection is broken because it hasn't attempted to flush the second half of the incomplete message.
The client makes a new connection and receives all messages from the beginning.
Repeat ...
There might be a bug in Servant somewhere since these connections are remaining open with un-flushed data, but I don't have time to track that down. A simple solution is to ensure that either (1) the server flushes its data to the socket or (2) the client waits for a full message before passing it to deserialization (framing). Ensuring the former (1) is preferrable to protect the server against using up file descriptors.
I've changed how our messages are framed to use NetstringFraming and I think it solves both (1) and (2), but I'm not sure. It may only be solving (2). I didn't use NewlineFraming because our ServerMessage type is encoded to a binary format using cereal which may contain newlines (unsure). In any case, please test this and tell me if you think it's satisfactory. It worked for me in my testing.
resolves #17
I think this will resolve the root cause of the error. My theory goes like this:
NoFraming
around messages.There might be a bug in Servant somewhere since these connections are remaining open with un-flushed data, but I don't have time to track that down. A simple solution is to ensure that either (1) the server flushes its data to the socket or (2) the client waits for a full message before passing it to deserialization (framing). Ensuring the former (1) is preferrable to protect the server against using up file descriptors.
I've changed how our messages are framed to use
NetstringFraming
and I think it solves both (1) and (2), but I'm not sure. It may only be solving (2). I didn't useNewlineFraming
because ourServerMessage
type is encoded to a binary format using cereal which may contain newlines (unsure). In any case, please test this and tell me if you think it's satisfactory. It worked for me in my testing.