nats-io / k8s

NATS on Kubernetes with Helm Charts
Apache License 2.0
445 stars 301 forks source link

Jetstream issue with fileStore disabled. #791

Closed samstride closed 1 year ago

samstride commented 1 year ago

Hi,

Thanks for maintaining the helm chart.

Running into the following issue on chart nats-1.0.2 and app 2.9.21

A config with fileStore disabled:

config:
  cluster:
    enabled: true
    replicas: 3

  jetstream:
    enabled: true

    memoryStore:
      enabled: true
      maxSize: 128Mi

    fileStore:
      enabled: false

podTemplate:
  topologySpreadConstraints:
    kubernetes.io/hostname:
      maxSkew: 1
      whenUnsatisfiable: ScheduleAnyway

Is causing the clients to throw this error:

nats: insufficient storage resources available

Things are back to normal once filestore is enabled.

Is this expected behaviour?

caleblloyd commented 1 year ago

What command are the clients executing?

If they are trying to make a file-backed stream then it's expected.

samstride commented 1 year ago

@caleblloyd , I am using the official go client.

       connectionOptions := []nats.Option{
        // Activate retry logic if a connection cannot be established at the start.
        nats.RetryOnFailedConnect(true),
        nats.ReconnectHandler(func(_ *nats.Conn) {
            // Note that this will be invoked for the first asynchronous connect.
            logger.Info("Connection to NATS restored ...")
        }),

       natsConnection, natsJetStream, err := NewJetStreamConnection(
        natsURL,
        connectionOptions,
        nil, // nil jetstream options
    )
    if err != nil {
        return nil, nil, err
    }

Hmmm, something must have changed.

jeromedoucet commented 1 year ago

@samstride

I don't now if that may be related with your problem, but the default storage type is, according to the documentation, the file one.

btw, the go client follow that :

const (
    // FileStorage specifies on disk storage. It's the default.
    FileStorage StorageType = iota
    // MemoryStorage specifies in memory only.
    MemoryStorage
)

I don't know how you manage your streams, but I guess you should explicitly set the memory storage type for them :shrug: .

samstride commented 1 year ago

@jeromedoucet . Thanks. Yes, looks like client now defaults to Filestorage.

It's interesting because previously clients did not need to specify it and nats would automatically handle things. So previously we had:

After client update, jetstream is defaulting to filestore. This means that we either:

I kinda prefer the old behaviour but I am sure there were reasons to move away from that.

Thanks.