search

eclipse / paho.mqtt.golang

Other
2.73k stars 533 forks source link

Possible memory leak #128

Closed Hades32 closed 6 years ago

Hades32 commented 7 years ago

After a week of very heavy usage (Several hundred clients. Each with exactly one subscription. Message rate ~1/sec/client) I see increasing memory usage.

Pprof shows this:

root@cf69f29f0647:/mrausc# go tool pprof http://xxxx-1:8086/debug/pprof/heap
Fetching profile from http://xxx-1:8086/debug/pprof/heap
Saved profile in /root/pprof/pprof.mqtt2redis.xxx-1:8086.alloc_objects.alloc_space.inuse_objects.inuse_space.001.pb.gz
Entering interactive mode (type "help" for commands)
(pprof) top25
872.97MB of 880.97MB total (99.09%)
Dropped 218 nodes (cum <= 4.40MB)
      flat  flat%   sum%        cum   cum%
  660.86MB 75.02% 75.02%   700.37MB 79.50%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.(*PublishPacket).Unpack
   94.59MB 10.74% 85.75%    94.59MB 10.74%  runtime.hashGrow
   53.50MB  6.07% 91.83%    53.50MB  6.07%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.NewControlPacketWithHeader
      48MB  5.45% 97.28%       48MB  5.45%  runtime.rawstringtmp
       7MB  0.79% 98.07%        7MB  0.79%  runtime.convT2E
    5.50MB  0.62% 98.69%     5.50MB  0.62%  runtime.makechan
    2.50MB  0.28% 98.98%    98.10MB 11.13%  runtime.mapassign
       1MB  0.11% 99.09%   754.87MB 85.69%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.ReadPacket
         0     0% 99.09%     8.50MB  0.96%  fmt.Sprintf
         0     0% 99.09%    97.09MB 11.02%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*MemoryStore).Put
         0     0% 99.09%     5.50MB  0.62%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1
         0     0% 99.09%   113.60MB 12.89%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.alllogic
         0     0% 99.09%    15.50MB  1.76%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.inboundKeyFromMID
         0     0% 99.09%   754.87MB 85.69%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.incoming
         0     0% 99.09%   112.60MB 12.78%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.persistInbound
         0     0% 99.09%    39.50MB  4.48%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.decodeString
         0     0% 99.09%   878.97MB 99.77%  runtime.goexit
         0     0% 99.09%       48MB  5.45%  runtime.slicebytetostring

(pprof) top25 -cumd command:
872.97MB of 880.97MB total (99.09%)
Dropped 218 nodes (cum <= 4.40MB)
      flat  flat%   sum%        cum   cum%
         0     0%     0%   878.97MB 99.77%  runtime.goexit
         0     0%     0%   754.87MB 85.69%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.incoming
       1MB  0.11%  0.11%   754.87MB 85.69%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.ReadPacket
  660.86MB 75.02% 75.13%   700.37MB 79.50%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.(*PublishPacket).Unpack
         0     0% 75.13%   113.60MB 12.89%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.alllogic
         0     0% 75.13%   112.60MB 12.78%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.persistInbound
    2.50MB  0.28% 75.41%    98.10MB 11.13%  runtime.mapassign
         0     0% 75.41%    97.09MB 11.02%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*MemoryStore).Put
   94.59MB 10.74% 86.15%    94.59MB 10.74%  runtime.hashGrow
   53.50MB  6.07% 92.22%    53.50MB  6.07%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.NewControlPacketWithHeader
      48MB  5.45% 97.67%       48MB  5.45%  runtime.rawstringtmp
         0     0% 97.67%       48MB  5.45%  runtime.slicebytetostring
         0     0% 97.67%    39.50MB  4.48%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho.mqtt.golang/packets.decodeString
         0     0% 97.67%    15.50MB  1.76%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.inboundKeyFromMID
         0     0% 97.67%     8.50MB  0.96%  fmt.Sprintf
       7MB  0.79% 98.47%        7MB  0.79%  runtime.convT2E
    5.50MB  0.62% 99.09%     5.50MB  0.62%  runtime.makechan
         0     0% 99.09%     5.50MB  0.62%  jira.moovel.com/stash/projects/TEL/mqtt2redis/vendor/github.com/eclipse/paho%2emqtt%2egolang.(*client).Connect.func1
alsm commented 7 years ago

Thanks for the report, I'll take a look into this, it may take a while to resolve.

janh commented 7 years ago

I'm using the file store and noticed that files are created for inbound messages but never deleted.

Looking at the code, inbound messages are deleted from the store in persistOutbound when called for the corresponding PUBACK/PUBCOMP message. However, it seems that persistOutbound isn't actually called anywhere except for PUBLISH messages.

Hades32 commented 7 years ago

That sounds pretty consistent with the pprof output above, as we can see there is quite some use of hashGrow...

Hades32 commented 7 years ago

@alsm any ideas for a fix? If I knew how to start I might have time for a PR...

Hades32 commented 6 years ago

Is there something we can do to help with this issue?

alsm commented 6 years ago

There was a PR just merged that addressed a memory leak, could you see if this has had any effect on your issue?

Hades32 commented 6 years ago

It definitely sounds like it! I'll re-open if it doesn't fix it :+1: