The is similar to #174 but I feel it's different enough to warrant its own issue.
We have a consumer in production running with client v1.0.6 and its memory usage is slowly increasing over time -- it starts at <100MB and I've seen it reach over 100 GB in 24 hours.
It seems that the vast majority of memory in use (93%) originates from github.com/bitly/go-nsq.(*Conn).readLoop.
Here is a profile:
$ go tool pprof --inuse_space bin-consumer mem.pprof
Entering interactive mode (type "help" for commands)
(pprof) top2
7.58GB of 8.14GB total (93.09%)
Dropped 143 nodes (cum <= 0.04GB)
Showing top 2 nodes out of 13 (cum >= 2.49GB)
flat flat% sum% cum cum%
5.09GB 62.46% 62.46% 7.58GB 93.10% github.com/bitly/go-nsq.(*Conn).readLoop
2.49GB 30.63% 93.09% 2.49GB 30.63% github.com/bitly/go-nsq.ReadResponse
Here's readLoop (I've removed irrelevant lines for brevity):
Counters based on incoming messages are incremented in-memory and flushed to the db once every 10 seconds.
Messages are not marked finished until the db flush has been done.
Each nsq message body is only 12 bytes + 26-byte message header, and peak usage is about 6000 messages per second. The math would indicate < 3 MB of message data being held in memory during those 10 seconds, waiting for the messages to be marked as finished.
The is similar to #174 but I feel it's different enough to warrant its own issue.
We have a consumer in production running with client v1.0.6 and its memory usage is slowly increasing over time -- it starts at <100MB and I've seen it reach over 100 GB in 24 hours.
It seems that the vast majority of memory in use (93%) originates from
github.com/bitly/go-nsq.(*Conn).readLoop
.Here is a profile:
Here's
readLoop
(I've removed irrelevant lines for brevity):And here's
ReadResponse
:In our nsq consumer:
Each nsq message body is only 12 bytes + 26-byte message header, and peak usage is about 6000 messages per second. The math would indicate < 3 MB of message data being held in memory during those 10 seconds, waiting for the messages to be marked as finished.
Any thoughts on what might be going on here?