Cannot unsubscribe after panic, alloc lots of memory when startup

[b04112106]

I'm using NATs Server in Go with lines below.

options := nserver.Options{
  Host:       host,
  Port:       port,
  StoreDir:   storeDir,
  JetStream:  true,
}
natsServer, err := nserver.NewServer(&options)
if err != nil {
  return nil, err
}
// Start NATS server in a goroutine
go natsServer.Start()

// .....
// ... serveral lines ...
// ......

natsConn, _ := nats.Connect(...)
js, _ := natsConn.JetStream()
consumer, _ := js.AddConsumer(SystemStream.Name, &consumerCfg)

After such operation, there would be a consumer under path like nats/jetStream/.../<stream_name>/obs/ When I delete the consumer, the consumer under the folder disappears as well.

However, if the process ends abnormally, the consumer under the path would never be removed.

Another issue is that when the jetStream contains lots of messages, it wastes time to recover those consumer.

Is there any method to remove unused consumers before they are recovered? Or I have to remove them after recovering has done?

[gcolliso]

Transferring to nats.go repo

[piotrpio]

Hey @b04112106, depending on what you need there are few options to achieve automatic cleanup:

• InactiveThreshold setting on ConsumerConfig - setting this will instruct the server to cleanup the consumer if it has been inactive for the specified duration. A consumer is considered inactive if there are no pending pull requests (for pull consumers) or to interest detected on deliver subject (for push consumers) - not if there are no messages to be delivered. For durable consumers, by default this is not set so the consumers will never get cleaned up.

• If you also want to prevent recovering consumer after server restart and you don't care about the messages you could also use a in-memory consumer by setting MemoryStorage on config. It is important to note that with in memory consumer it will be lost as soon as your nats-server process is terminated.

Let me know if this answers your question.

[b04112106]

Hi @piotrpio , thank you for the recommandation!! It helps a lot and is precisely what I want. I choose to use memory storage but am afraid of running out of memory. The size of my jetstream is about 14GB and # of consumers is about 6. When I try to AddConsumer, the memory usage climbs up to 6GB. Is it normal? I'm still trying to avoid such situation.

update: after AddConsumer I use

js.PullSubscribe("", "", nats.Bind(streamName, consumerName))

And I found that AddConsumer itself allocate lots of memory.

[Jarema]

Consumers does not take much space themselves. They do not store messages. Stream does. However, in Interests based streams, consumers can affext number of messages in the stream.

Can you share your stream config? We would also need info about stream content to answer the question if it's normal to have few gigabyes of memory allocated.

[b04112106]

Hi @Jarema , thank you for the reply and here is my stream config!

SystemStream = nats.StreamConfig{
        Name:     name,
        Subjects: []string{name + ".>"},
    }

The stream content is a struct generated by protobuf. The size of message varies from 10^1 KB to 160 KB and it depends on the length of slice. I've already had 1,400,000 messages and the total size is 14 gigabytes.

I use pyroscope to monitor the usage of memory and found that if I only call go natsServer.Start() to run the nats server, the size of memory allocated does not grow rapidly. However, after I call AddConsumer or Fetch, allocated memory grows. The peak is 22 GB over 32 GB.

[Jarema]

What verison of nats-server are you running?

The memory usage depends a lot on traffic, cluster setup, and a lot of other factors. Could you share some context on your setup?

I realized I didn't answer how to manage consumers. You should not delete your JetStream files, but rather delete the consumer from API.

[b04112106]

The version of nats-server is 2.10.0. The server is run in container while the client connect from the same container as well. Actually, client connects to localhost.

I didn't delete JetStream files nor consumers. I call AddConsumer after the connection is established. There are about 30 consumers. I just doubt that because all of these consumer needs to fetch message from the JetStream and the amount of allocated memory would only go down after gc mechanism of golang is triggerred. Does it make sense?

[b04112106]

After some experiments, we found that the memory usage peak is due to repeatedly AddConsumer Failed. In our implementation, we try to AddConsumer in our Service. Service could be considered a for-loop to run a func() error and will restart when the function returns an error. The interval between the startup and the end of service varies from 0.1 seconds to 102.4 seconds. Each time the service function fails, we double the interval and up to 102.4 seconds.

The large size of JetStream seems to increase the time consumption of AddConsumer. Our new confusion is

1. Why does it take so much memory to AddConsumer?
2. Why does it NOT release memory after context deadline exceeded?
3. If I add Durable in consumer config, would it possibly be the solution to this issue?