Secrets permission issue with the custom security context

[ajurcenk]

What would you like to be added?

When the custom security context is set in the helm charts version: 5.8.12:

securityContext:
    fsGroup: 101
    runAsUser: 101
    runAsNonRoot: true
    allowPriviledgeEscalation: false

and tls is enabled, the error occurs starting the redpanda pod: RROR 2024-08-05 14:40:11,031 [shard 0:main] main - application.cc:461 - Failure during startup: std::__nested<std::runtime_error> (Could not read trust file /etc/tls/certs/default/ca.crt): std::__1::__fs::filesystem::filesystem_error (error system:13, filesystem error: open failed: Permission denied ["/etc/tls/certs/default/ca.crt"]

The generated stateful set for the redpanda for the secret:

- name: redpanda-default-cert
        secret:
          defaultMode: 288
          secretName: redpanda-default-cert 

Thhe defaultMode:288 gives read permissions to the owner and the group but no permissions to others. For other's secrets, the deafultMode:509 permission.

Why is this needed?

To support the custom security context.

JIRA Link: K8S-310

[chrisseto]

I'm having trouble reproducing this issue. Do you have an exact values file you could share? There's no top level securityContext value so I've been testing with:

statefulset:
  securityContext:
      fsGroup: 101
      runAsUser: 101
      runAsNonRoot: true
      allowPriviledgeEscalation: false

and

statefulset:
  podTemplate:
    spec:
      securityContext:
        fsGroup: 101
        runAsUser: 101
        runAsNonRoot: true
        allowPriviledgeEscalation: false

Both are working as expected but I do see how it might be possible to get such a result. The chart is a bit wonky to match historical behaviors and it defaults runAsGroup to the value of fsGroup which is why this works in the cases shown and the defaults. However, if fsGroup != runAsUser or runAsGroup wasn't getting set for some reason, you could certainly see such an error.

[chrisseto]

@ajurcenk any update here?

[RafalKorepta]

I can only add that tuning init container needs to be drop from statefulset with the following helm values:

tuning:
  tune_aio_events: false

The rest of values provided from @chrisseto should be concatenated.

[david-yu]

@ajurcenk Will go ahead and close as this is not reproducible.