nats-io / jsm.go

JetStream Management Library for Golang
Apache License 2.0
151 stars 26 forks source link

Nil pointer #455

Open Davincible opened 1 year ago

Davincible commented 1 year ago

I'm getting a nil pointer on streams.go:176. Not sure why. Maybe invalid stream/subject

ripienaar commented 1 year ago

What version?

please show the stack trace.

Davincible commented 1 year ago

I'm sorry, didn't save the full stack trace & can't find it anymore. Moved on to using the new nats-io/jetstream package.

Version was latest, I checked and it was this line; https://github.com/nats-io/jsm.go/blob/ef62e6a49a6b308c20b94e16b15079b26182ced2/streams.go#L176. Info must have been nil since stream is accessed earlier

Code I used ```go func (s *StreamCacheStorage[T]) listStreamMessages(subj string) ([]T, error) { if len(subj) == 0 { return []T{}, nil } stream, err := s.jsmM.LoadStream(s.streamName) if err != nil { return nil, fmt.Errorf("load stream: %w", err) } pager, err := stream.PageContents( jsm.PagerFilterSubject(subj), jsm.PagerSize(batchLimit), ) if err != nil { return nil, fmt.Errorf("page content: %w", err) } var msgs []*nats.Msg hasMore := true for hasMore { msg, last, err := pager.NextMsg(context.Background()) if !last && err != nil { return nil, fmt.Errorf("fetch next stream page: %w", err) } msgs = append(msgs, msg) hasMore = !last } return unmarshalMsgs[T](msgs) } ```

Since the consumers were not being deleted, I ended up with 20k something consumers, that might have had something to do with it. Only showed up at scale in prod system. Local testing went fine

ripienaar commented 1 year ago

Why were consumers not deleted? Consumers are set to auto remove after a hour of idle time, are you running a particularly old server?

https://github.com/nats-io/jsm.go/blob/a21621a6f5fbccde770b4c268c83aed60bf5e683/stream_pager.go#L169

ripienaar commented 1 year ago

Apart from that, I would not use stream pager for what you are doing. Make a consumer and just fetch the messages. This repo is mainly orientated to needs of the CLI/Terraform/GHA features, you should use nats.go instead.

Davincible commented 1 year ago

It was a new server. Initially used this lib as I couldn't get what I wanted with main repo, and looked at the CLI for an example. I do have it working now with the new jetstream package though, not using this code anymore.