Enabling sasl after initial cluster is formed breaks Schema Registry

voutilad commented 5 months ago

What happened?

Using the Helm chart, adding something like:

auth:
  sasl:
    enabled: true
    secretRef: "redpanda-superuser"
    users: []

And deploying the update will enable SASL and authorization for all listeners, but the Schema Registry and any internal Kafka client usage will not properly use authentication. This results in any client (Console, rpk, etc.) trying to use the Schema Registry to receive errors about it being unable to contact brokers:

Broker side:

WARN  2024-03-26 07:09:11,106 [shard 0:main] pandaproxy - service.cc:364 - mitigate_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })

Client side:

{"error_code":50302,"message":"{ node: -1 }, { error_code: broker_not_available [8] }"}

What did you expect to happen?

Schema Registry to work 😆

How can we reproduce it (as minimally and precisely as possible)?. Please include values file.

Start with:

```yaml image: repository: docker.redpanda.com/redpandadata/redpanda tag: v23.3.9 pullPolicy: Always resources: cpu: cores: 1 overprovisioned: true memory: container: min: 2Gi max: 2Gi tls: enabled: true statefulset: budget: maxUnavailable: 1 initContainers: setDataDirOwnership: enabled: true external: enabled: true service: enabled: false addresses: - redpanda-broker-0 - redpanda-broker-1 - redpanda-broker-2 ```

Then update to:

```yaml image: repository: docker.redpanda.com/redpandadata/redpanda tag: v23.3.9 pullPolicy: Always resources: cpu: cores: 1 overprovisioned: true memory: container: min: 2Gi max: 2Gi tls: enabled: true statefulset: budget: maxUnavailable: 1 initContainers: setDataDirOwnership: enabled: true external: enabled: true service: enabled: false addresses: - redpanda-broker-0 - redpanda-broker-1 - redpanda-broker-2 auth: sasl: enabled: true secretRef: "redpanda-superuser" users: [] ```

Anything else we need to know?

No response

Which are the affected charts?

Redpanda

Chart Version(s)

5.7.35

Cloud provider

n/a

JIRA Link: K8S-127

BenPope commented 5 months ago

And deploying the update will enable SASL and authorization for all listeners

Is this true? In the cluster config I saw, enable_sasl and kafka_enable_authorization were both set to false.

The helm chart may have updated bootstrap.yaml and redpanda.yaml, but after a cluster is formed, cluster config from those two locations is ignored.

The cluster config need to be set with something like:

rpk cluster config set kafka_enable_authorization true

vuldin commented 5 months ago

I think this issue is also related.

mattschumpert commented 5 months ago

Good catch, this does seem like a broken workflow

Heads up @chrisseto . These changes from the helm side to existing cluster configs must definitely set the cluster configs via RPK or Admin API and trigger a restart when necessary. Node level configs could still go into redpanda.yaml and trigger a restart IIUC

redpanda-data / helm-charts