nats-io / k8s

NATS on Kubernetes with Helm Charts
Apache License 2.0
446 stars 302 forks source link

can't create nats jetstream with memory only #861

Closed justinrush closed 7 months ago

justinrush commented 7 months ago

What version were you using?

1.1.7

What environment was the server running in?

kubernetes helm chart

Is this defect reproducible?

Yes. You can slim it down further, but my values overrides are below. Basically I'm trying to deploy nats with only memory storage and 0 file.

container:
  merge:
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        memory: 1Gi
global:
  image:
    registry: artifactory.epic.com/docker-remote
tlsCA:
  enabled: true
  secretName: cm-issued-nats-ephemeral-data-client-tls
  dir: /etc/nats-ca-cert
  key: ca.crt
natsBox:
  contents:
    default:
      tls:
        secretName: cm-issued-nats-ephemeral-data-client-tls
        cert: tls.crt
        key: tls.key
config:
  merge:
    authorization:
      users:
        - user: mars-rover
  cluster:
    enabled: true
    replicas: 3
    tls:
      enabled: true
      secretName: cm-issued-nats-ephemeral-data-server-tls
  jetstream:
    enabled: true
    fileStore:
      enabled: false
    memoryStore:
      enabled: true
      maxSize: 128Mi
  nats:
    tls:
      enabled: true
      secretName: cm-issued-nats-ephemeral-data-client-tls
      cert: tls.crt
      key: tls.key
      # https://docs.nats.io/running-a-nats-service/configuration/securing_nats/auth_intro/tls_mutual_auth#mapping-client-certificates-to-a-user
      # checks if any SAN maps to a user
      merge:
        verify_and_map: true
promExporter:
  enabled: true

The resulting nats config:

{
  "authorization": {
    "users": [
      {
        "user": "mars-rover"
      }
    ]
  },
  "cluster": {
    "name": "nats-ephemeral",
    "no_advertise": true,
    "port": 6222,
    "routes": [
      "tls://nats-ephemeral-0.nats-ephemeral-headless:6222",
      "tls://nats-ephemeral-1.nats-ephemeral-headless:6222",
      "tls://nats-ephemeral-2.nats-ephemeral-headless:6222"
    ],
    "tls": {
      "ca_file": "/etc/nats-ca-cert/ca.crt",
      "cert_file": "/etc/nats-certs/cluster/tls.crt",
      "key_file": "/etc/nats-certs/cluster/tls.key"
    }
  },
  "http_port": 8222,
  "jetstream": {
    "max_file_store": 0,
    "max_memory_store": 128Mi
  },
  "lame_duck_duration": "30s",
  "lame_duck_grace_period": "10s",
  "pid_file": "/var/run/nats/nats.pid",
  "port": 4222,
  "server_name": $SERVER_NAME,
  "tls": {
    "ca_file": "/etc/nats-ca-cert/ca.crt",
    "cert_file": "/etc/nats-certs/nats/tls.crt",
    "key_file": "/etc/nats-certs/nats/tls.key",
    "verify_and_map": true
  }
}

Given the capability you are leveraging, describe your expectation?

I expect to be able to use memory only storage without any file at all, ephemeral or otherwise.

Given the expectation, what is the defect you are observing?

The nats pods say this when attempting to create a keyvalue:

nats [7] 2024/01/24 21:27:52.331836 [WRN] JetStream failed to lookup running stream while removing stream '$G > KV_mars' from this server

my client says this:

creating keyvalue: nats: API error: code=500 err_code=10047 description=insufficient storage resources available
justinrush commented 7 months ago

this is my bad - default storage type when creating keyvalue is file; specifying memory will fix that.

caleblloyd commented 7 months ago

Great! Feel free to drop into the #k8s channel on https://slack.nats.io/ also, can help with stuff like this a little faster in there usually!