Nats Streaming operator didn't update cluster size

[exu]

I'm trying to install / update nats-streaming operator with postgresql attached as storage - and I've recently get a problem with not responding to changes in NatsStreamingCluster

$ k get NatsStreamingCluster -o yaml

apiVersion: v1
items:
- apiVersion: streaming.nats.io/v1alpha1
  kind: NatsStreamingCluster
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"streaming.nats.io/v1alpha1","kind":"NatsStreamingCluster","metadata":{"annotations":{},"name":"nats-streaming","namespace":"default"},"spec":{"configFile":"/etc/stan/config/secret.conf","natsSvc":"nats","size":2,"store":"SQL","template":{"spec":{"containers":[{"name":"nats-streaming","volumeMounts":[{"mountPath":"/etc/stan/config","name":"stan-secret","readOnly":true}]}],"volumes":[{"name":"stan-secret","secret":{"secretName":"stan-secret"}}]}}}}
    clusterName: ""
    creationTimestamp: "2019-02-26T11:42:42Z"
    generation: 1
    name: nats-streaming
    namespace: default
    resourceVersion: "51146622"
    selfLink: /apis/streaming.nats.io/v1alpha1/namespaces/default/natsstreamingclusters/nats-streaming
    uid: a0426fe2-39bb-11e9-b4aa-42010a9c00b8
  spec:
    configFile: /etc/stan/config/secret.conf
    natsSvc: nats
    size: 2
    store: SQL
    template:
      spec:
        containers:
        - name: nats-streaming
          volumeMounts:
          - mountPath: /etc/stan/config
            name: stan-secret
            readOnly: true
        volumes:
        - name: stan-secret
          secret:
            secretName: stan-secret
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

But after installing there is only one pod of nats-streaming

namespace is set to 'default'
NAME                                      READY   STATUS    RESTARTS   AGE
nats-1                                    1/1     Running   0          21m
nats-2                                    1/1     Running   0          21m
nats-3                                    1/1     Running   0          21m
nats-operator-56c4974b7f-wsp6j            1/1     Running   0          21m
nats-streaming-1                          1/1     Running   0          15m
nats-streaming-operator-d5cbd6665-8k6rt   1/1     Running   0          21m

After editing resource and setting size to different value (4)

 k edit NatsStreamingCluster

operator didn't responds to events.

info

!! There is only one instance when SQL config used - with default installation with file store there is valid number of instances !!

Env: GKE

[exu]

I don't have any knowledge about wrtiting operators but I've found comment in in Kubernetes go client - maybe it'll help a little

// NewIndexerInformer returns a Indexer and a controller for populating the index
// while also providing event notifications. You should only used the returned
// Index for Get/List operations; Add/Modify/Deletes will cause the event
// notifications to be faulty.
//

it's used here https://github.com/nats-io/nats-streaming-operator/blob/master/internal/operator/controller.go#L145

[wallyqs]

Hi @exu this is because when setting the store to be SQL then only need a single instance from the service will be required. DB mode is not compatible with clustered mode so can't have multiple instances: https://github.com/nats-io/nats-streaming-operator#using-a-db-store Since the server does not hold any state it is good enough for Kubernetes to just restart the streaming service.

[exu]

Hmm Thanks for pointing this out, I understand, but what about scaling? It looks like candidate for bottleneck during high traffic.

[wallyqs]

yes that is also a current limitation, for scaling out you could use nats streaming server instances for different purposes, but right now clustering is mostly for HA when using a file store (https://github.com/nats-io/nats-streaming-server/issues/494)