nats-io / nack

NATS Controllers for Kubernetes (NACK)
Apache License 2.0
160 stars 61 forks source link

NACK gave up on creating a stream before NATS became available #193

Open rozidan opened 4 months ago

rozidan commented 4 months ago

What version were you using?

NACK: 0.15.0 Nats-server: 2.10.9

What environment was the server running in?

RKE2: v1.26.4+rke2r1

Is this defect reproducible?

Before the NATS server is available, start Nack and let it attempt to create a stream more then ten times.

Given the capability you are leveraging, describe your expectation?

The stream should be recreated

Would it be possible for the "maximum queue retries" to be configurable? See https://github.com/nats-io/nack/blob/38b4b81470430a1a4a02c741fae6156c8687b259/controllers/jetstream/controller.go#L57

Given the expectation, what is the defect you are observing?

The stream is not created nack log: "controller.go:478] failed to process stream: failed to create stream "...": no suitable peers for placement (10005)"

jetdream commented 2 months ago

+1 The same issue but is happening even earlier - when NATS cluster is not deployed yet:

apiVersion: jetstream.nats.io/v1beta2
kind: Stream
# skipped...
status:
  conditions:
    - lastTransitionTime: '2024-08-29T03:14:17.636645001Z'
      message: >-
        failed to connect to nats-servers(nats://nats:4222): dial tcp:
        lookup nats on 10.245.0.10:53: no such host
      reason: Errored
      status: 'False'
      type: Ready
  observedGeneration: 0
Jarema commented 2 months ago

We plan to have a prpoer control loop that would make it an ongoing operation instaed of single one. I think that is what casuses the issue now - it just tries once.