RBAC errors seen while RBAC synchronizes on cluster

ocofaigh commented 3 years ago

IBM Internal Project Golden Eye

When a cluster is provisioned, and accessed for the first time by a user, you see the message: If you are accessing the cluster for the first time, 'kubectl' commands might fail for a few seconds while RBAC synchronizes.

We have seen an issue where the data lookup using ibm_container_cluster_config on a newly provisioned cluster, can give errors like:

Error from server (Forbidden): deployments.apps "downloads" is forbidden: User "IAM#goldeneye.development@ibm.com" cannot get resource "deployments" in API group "apps" in the namespace "openshift-console"

configmaps "cluster-proxy-config" is forbidden: User "IAM#goldeneye.development@ibm.com" cannot get resource "configmaps" in API group "" in the namespace "default"

We are working around this right now by actually attempting to run a command against the kube api server (kubectl get namespaces) in a retry loop until it passes. See https://github.ibm.com/GoldenEye/kubeconfig-module/blob/master/main.tf

Is it possible to add this into the provider code in the relevant place? Maybe in the code that does the data lookup?

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

$ terraform -v
Terraform v1.0.8
on darwin_amd64
+ provider registry.terraform.io/hashicorp/external v2.1.0
+ provider registry.terraform.io/hashicorp/helm v2.3.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.5.0
+ provider registry.terraform.io/hashicorp/local v2.1.0
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.7.2
+ provider registry.terraform.io/ibm-cloud/ibm v1.32.1

Affected Resource(s)

ibm_container_cluster_config

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please share a link to the ZIP file.

Debug Output

Panic Output

Expected Behavior

Actual Behavior

Steps to Reproduce

terraform apply

Important Factoids

References

0000

kavya498 commented 3 years ago

Is this an intermittent issue or Do you see this error every time you create a new cluster?

ocofaigh commented 3 years ago

It is intermittent - I guess it is a timing issue. Here is an example of the workaround actually doing its job..

TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace: Creating...
TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace: Provisioning with 'local-exec'...
TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace (local-exec): Executing: ["/bin/bash" "-c" "../../scripts/check_rbac_synchronized.sh"]
TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace (local-exec): Attempting to get namespaces to ensure RBAC fully synchronized...
TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace (local-exec): Error from server (Forbidden): namespaces is forbidden: User "IAM#goldeneye.development@ibm.com" cannot list resource "namespaces" in API group "" at the cluster scope
TestTerraformKubeconfig 2021-10-04T23:29:45Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace (local-exec): Failed to get namespaces, retrying in 10s. (Attempt 1/10)
TestTerraformKubeconfig 2021-10-04T23:29:55Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace: Still creating... [10s elapsed]
TestTerraformKubeconfig 2021-10-04T23:29:55Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace (local-exec): Successfully retrieved namespaces, RBAC synchronization complete
TestTerraformKubeconfig 2021-10-04T23:29:55Z logger.go:66: module.kubeconfig.null_resource.check_get_namespace: Creation complete after 11s [id=897635755818830043]

As you can see on the first attempt, we hit the issue, and on second attempt it passed. So I think we are only talking about a few seconds in general.

kavya498 commented 3 years ago

okay.. We shall look into it..

IBM-Cloud / terraform-provider-ibm