Unable to see existing users & policies on Tenant Console UI

Sanketbhandare commented 1 month ago

Expected Behavior

All existing users & policies should be visible on Tenant Console UI

Current Behavior

Unable to see any existing users & policies on Tenant Console UI

Possible Solution

Creation of user & policy via mc CLI command and attach / detach of policy to user based on user role & requirement.

Steps to Reproduce (for bugs)

Not sure on how to reproduce this issue. Out of 3 env, 2 env are facing this issue. We've haven't performed any major activity on those cluster.

Context

Not sure on how to reproduce this issue. Out of 3 env, 2 env are facing this issue. We've haven't performed any major activity on those cluster. We're using Longhorn as backend storage for Minio and Hashicorp Vault as KES Server secret store.

Below error logs observed in Tenant Pod:

DeploymentID: 90095a7f-3c30-4546-b223-923d848c4a75
Error: Unable to initialize config, some features may be missing: decryption failed: ciphertext is not authentic (*fmt.wrapError)
       7: internal/logger/logger.go:259:logger.LogIf()
       6: cmd/logging.go:102:cmd.configLogIf()
       5: cmd/server-main.go:596:cmd.initConfigSubsystem()
       4: cmd/server-main.go:561:cmd.initServerConfig()
       3: cmd/server-main.go:845:cmd.serverMain.func11()
       2: cmd/server-main.go:522:cmd.bootstrapTrace()
       1: cmd/server-main.go:844:cmd.serverMain()

API: SYSTEM.iam
Time: 07:13:33 UTC 10/09/2024
DeploymentID: 90095a7f-3c30-4546-b223-923d848c4a75
Error: IAM sub-system is partially initialized, unable to write the IAM format: decryption failed: ciphertext is not authentic (*fmt.wrapError)
       6: internal/logger/logger.go:259:logger.LogIf()
       5: cmd/logging.go:18:cmd.iamLogIf()
       4: cmd/iam.go:306:cmd.(*IAMSys).Init()
       3: cmd/server-main.go:874:cmd.serverMain.func12.1()
       2: cmd/server-main.go:522:cmd.bootstrapTrace()
       1: cmd/server-main.go:873:cmd.serverMain.func12()
MinIO Object Storage Server
Copyright: 2015-2024 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2024-05-01T01-11-10Z (go1.21.9 linux/amd64)

API: https://minio.dev-minio.svc.cluster.local 
WebUI: https://10.244.243.56:9443 https://127.0.0.1:9443   

Docs: https://min.io/docs/minio/linux/index.html
Status:         1 Online, 0 Offline. 
STARTUP WARNINGS:
- The standard parity is set to 0. This can lead to data loss.

Regression

Your Environment

Version used (minio-operator): v5.0.15
Environment name and version (e.g. kubernetes v1.17.2): v1.28.2
Minio & mc version: bash-5.1$ minio --version minio version RELEASE.2024-05-01T01-11-10Z (commit-id=7926401cbd5cceaacd9509f2e50e1f7d636c2eb8) Runtime: go1.21.9 linux/amd64 License: GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html Copyright: 2015-2024 MinIO, Inc.

bash-5.1$ mc --version mc version RELEASE.2024-04-29T09-56-05Z (commit-id=5b7b2223717a32ff01d63d57c2d040a719ca581e) Runtime: go1.21.9 linux/amd64 Copyright (c) 2015-2024 MinIO, Inc. License GNU AGPLv3 https://www.gnu.org/licenses/agpl-3.0.html

Operating System and version (uname -a): Linux devk8swkr1 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Thu Jan 12 10:05:00 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
Link to your deployment file:

ramondeklein commented 1 month ago

It looks like there is an encryption failure, so IAM cannot read anything. It looks like something is wrong with your KES configuration or back-end. Which KES provider are you using? If you store KES keys on ephemeral storage, then this is to be expected when you recreate the pod...

Sanketbhandare commented 1 month ago

@ramondeklein As mentioned earlier in the issue, we're using Hashicorp vault as KES provider to store keys. KES keys are stored vault and are accessible. In one of our test environment, we've tried restarting tenant pod couple of times, and observed no issues.

If you refer the tenant logs mentioned above, it seems that something wrong in the Tenant backend, for IAM user / policy. Unable to understand the root cause for this issue.

Error: Unable to initialize config, some features may be missing: 
decryption failed: ciphertext is not authentic (*fmt.wrapError)

Error: IAM sub-system is partially initialized, unable to write the IAM format: 
decryption failed: ciphertext is not authentic (*fmt.wrapError)

ramondeklein commented 1 month ago

When decryption fails, then it is typically a key failure. Did you rotate the key in Hashicorp Vault?

Sanketbhandare commented 1 month ago

@ramondeklein Nope, we don't rotate key in Hashicorp vault.

aead commented 1 month ago

@Sanketbhandare Which KES version are you using. Have you updated the KES image or updated the KES config secret? Are the KES image versions identical across all pods and did you downgrade KES from a newer to an older version at any point in time.

Sanketbhandare commented 1 month ago

Which KES version are you using ? -> kes:2024-04-12T13-50-00Z Have you updated the KES image or updated the KES config secret? -> We haven't updated KES image, but KES config secret is updated due to Vault migration in our env. Secret which was updated was kes-configuration and we've replaced the the value of approle id & secret from vault.

We have other environments as well, but no migration is done in those env, but still we are observing this issue. We are unable to see default policies.

minio / operator