Disconnects seem to be quite ungraceful

[karalabe]

Been playing around with NSQ a lot lately and I keep hitting walls when trying to write test suites for assembling various network topologies. Most of the issues seems to stem from NSQD not handling properly consumer disconnects (I'm using go-nsq). I don't even know where to describe the strange things:

• When stopping a consumer, sometimes the CLS message is sent to NSQD, sometimes it is not.
• Even if the CLS does get to NSQD, sometimes it seems to not respond with CLOSE_WAIT, rather nukes the stream.
• The logs are full of error messages on both consumer and broker side during shutdowns that one side or another tries to read/write but the stream is already dead (no graceful disconnect).
• Disconnecting the last consumer doesn't seem to decrement the client count of a topic/channel.
• Disconnecting a consumer doesn't seem to abort/reschedule the in-flight messages for that consumer.

Seems to me that the entire shutdown pathway is very very wrong, just that various timeouts hack around the root cause. E.g. the client heartbeats (or lack thereof after a disconnect) is the one that will trigger the cleanup of leftover client counts; the in-flight timeout is the one that reschedules messages nor processed by a disconnected client.

I'm unsure if I'm doing something weird here, but it seems that NSQ is very very prone to weird behavior when I have very short lived connections.

[ploxiln]

Honestly, we haven't historically worried much about clean client disconnects, and our tests don't pay attention to that in particular (just that messages go where they should go). We have a few existing issues about noisy logs related to as-clean-as-currently-possible disconnects ...

https://github.com/nsqio/nsq/issues/521 https://github.com/nsqio/nsq/pull/582 https://github.com/nsqio/go-nsq/issues/103

it seems that NSQ is very very prone to weird behavior when I have very short lived connections

That is plausible, it was not designed for short lived tcp-protocol connections. But if you can offer some good fixes/cleanups for these cases, that would be great :)

[mreiferson]

I'm inclined to move this issue over to go-nsq as I think it is likely the major contributing factor here.