mochi-mqtt / server

The fully compliant, embeddable high-performance Go MQTT v5 server for IoT, smarthome, and pubsub
MIT License
1.22k stars 212 forks source link

memory problem #429

Open jeasonHu opened 1 month ago

jeasonHu commented 1 month ago

In the process of testing and verifying, found some memory problem; The following are the test steps 1. start server in k8s with 2 pod 2. Initiate 2000 connection requests,when execution is complete close test client 3. few minutes later Initiate 3000 connection requests,when execution is complete close test client 4. repeat step 3

The memory usage: image

After performing steps 23 Test client has close, all clients has disconnected,bus memory not recovery enough ,causing a backlog; then exec steps 4 the memory exceeded the limit, causing the service to restart.

Code: github.com/mochi-mqtt/server/v2 v2.6.5

` capabilities := mqtt.NewDefaultServerCapabilities() capabilities.MaximumSessionExpiryInterval = 60 capabilities.MaximumInflight = 1024 capabilities.MaximumClientWritesPending = 256

mqttServer = mqtt.New(&mqtt.Options{
    Listeners:                nil,
    Hooks:                    nil,
    Capabilities:             capabilities,
    ClientNetWriteBufferSize: 0,
    ClientNetReadBufferSize:  0,
    Logger:                   logger,
    SysTopicResendInterval:   5,
    InlineClient:             true,
})`

I'm not sure what caused this issue. I made some configuration changes based on the example code.

thedevop commented 1 month ago

@jeasonHu, few things:

  1. are your clients MQTTv3 or v5?
  2. can you verify these clients between step 3 and 4 in fact disconnected from the server? You can check the SysInfo.
  3. what's the value set for MaximumMessageExpiryInterval?

Note, setting ClientNetWriteBufferSize and ClientNetReadBufferSize to 0 equals to 2KB.

jeasonHu commented 1 month ago

@jeasonHu, few things:

  1. are your clients MQTTv3 or v5?
  2. can you verify these clients between step 3 and 4 in fact disconnected from the server? You can check the SysInfo.
  3. what's the value set for MaximumMessageExpiryInterval?

Note, setting ClientNetWriteBufferSize and ClientNetReadBufferSize to 0 equals to 2KB.

@thedevop

  1. My client version is v3.
  2. I have confirmed that both clients_connected and clients_total in sysinfo have returned to their initial states.
  3. I set MaximumMessageExpiryInterval to 60, and configured ClientNetWriteBufferSize and ClientNetReadBufferSize to 1024.
thedevop commented 1 month ago

If both MaximumMessageExpiryInterval and MaximumSessionExpiryInterval are set to 60, the server should release all the memory used by the clients (however Go may not release the memory back to OS).

There are 2 other possibilities:

  1. Do you have any custom hooks that's holding on the the client?
  2. For efficiency reason, Go does not release the freed memory back to OS immediately, you can fine tune this following the GC-guide.