engine Kubernetes as default stop woking after 0.22.0

hornv commented 1 year ago

Bug describtion

I have a following configuration for helm chart for kubernetes

    root: disabled
    admin:
      identity: disabled
    log:
      error: on
      audit: on
    tls:
      key:  /etc/kes-tls/tls.key
      cert: /etc/kes-tls/tls.crt
    policy:
      minio:
        allow:
        - /v1/key/create/minio*
        - /v1/key/generate/minio*
        - /v1/key/decrypt/minio*
        identities:
        - ${MINIO_IDENTITY}
    keystore:
      vault:
        endpoint: http://vault.vault:8200
        kubernetes:
          engine: ""
          role: "kes"
          jwt:  "/var/run/secrets/kubernetes.io/serviceaccount/token"
          retry:
        status:
          ping: 10s

The values are being provisioned by init container. This configuration worked totally fine up to v0.21.1 But version 0.22.0 and higher start bringing the error

Error: Error making API request.

URL: PUT http://vault.vault:8200/v1/auth/kubernetes/login
Code: 403. Errors:

* permission denied

I am pretty sure that problem connected to this PR https://github.com/minio/kes/commit/87de9e00d3dbf7b111f77787e6cd0381ffcc2ccd

Expected behavior

it should recreate pod and run it. but I have CrashLoopBackOff only

Additional context

I have kubernetes (k3s) deployment based on helm charts.

aead commented 1 year ago

Hi @hornv thanks for reporting this issue. I don't think this issue is related to 87de9e0 since the Vault client (in v0.21.1 and later) still pick the authentication method based on the presence of an approle id & secret resp. a K8S role & JWT. Ref: https://github.com/minio/kes/blob/v0.21.1/internal/vault/vault.go#L93-L102

However, I think this issue is caused by the JWT being not read from the file you provided. On v0.21.1, we check whether the config value is a file and if so, read the JWT from disk. Ref: https://github.com/minio/kes/blob/a91227c90019608b601a08c5883bb8dc42355aa8/internal/yml/config.go#L77

However, this seems to have changed. I have to confirm this theory, tough. Will get back with a confirmation.

hornv commented 1 year ago

Thank you for your attention. I just thought that after that PR you started getting the default value from approle.engine which I have empty. But never mind. it was only guess. let me know ,if you need any additional info

minio / kes