marianleica / azrez

Azure Resource Launcher (azrez) is a console application, created by @marianleica, that aims to support those looking for quick deployments in Azure VMs and Kubernetes for learning and swift repros.
https://github.com/marianleica/azrez
MIT License
1 stars 0 forks source link

Build Scenario 5.1/5.2/5.3 - Running Kubectl Commands after deployment Error #14

Open Andu1391 opened 4 days ago

Andu1391 commented 4 days ago

Tested on CloudShell / AzureCLI and Run Command. First Try:

Deployment - ok Onboarding to ARC, encountered error below.

Onboarding cluster aks-kubenetlb-3412 to Azure Arc-enabled Kubernetes Default enabled including preview versions for extension installation now. Disabled in future release. Use '--allow-preview true' to enable it specifically if needed. Use '--allow-preview false' to install stable version only. Extension 'connectedk8s' 1.9.2 is already installed. The command failed with an unexpected error. Here is the traceback: No module named 'azure.cli.core.auth.msal_authentication' Traceback (most recent call last): File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 564, in execute File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/init.py", line 526, in load_arguments File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/init.py", line 321, in load_arguments File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/commands.py", line 104, in load_arguments File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 125, in arguments_loader File "D:\a_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 59, in get_op_handler File "importlib__init__.py", line 126, in import_module File "", line 1204, in _gcd_import File "", line 1176, in _find_and_load File "", line 1147, in _find_and_load_unlocked File "", line 690, in _load_unlocked File "", line 940, in exec_module File "", line 241, in _call_with_frames_removed File "C:\Users\alexandrupop.azure\cliextensions\connectedk8s\azext_connectedk8s\custom.py", line 47, in import azext_connectedk8s._clientproxyutils as clientproxyutils File "C:\Users\alexandrupop.azure\cliextensions\connectedk8s\azext_connectedk8s_clientproxyutils.py", line 21, in from azure.cli.core.auth.msal_authentication import ServicePrincipalCredential ModuleNotFoundError: No module named 'azure.cli.core.auth.msal_authentication' To check existing issues, please visit: https://github.com/Azure/azure-cli/issues

• Local system Info: 

PS C:\azrez> az version { "azure-cli": "2.65.0", "azure-cli-core": "2.65.0", "azure-cli-telemetry": "1.1.0", "extensions": { "account": "0.2.5", "arcappliance": "1.1.1", "connectedk8s": "1.9.2", "ssh": "2.0.3" } }

PS C:\azrez> az upgrade PS C:\azrez> az version { "azure-cli": "2.65.0", "azure-cli-core": "2.65.0", "azure-cli-telemetry": "1.1.0", "extensions": { "account": "0.2.5", "arcappliance": "1.3.0", "connectedk8s": "1.10.1", "ssh": "2.0.6" } }

• New Try: 

Deployment - ok Onboarding - ok

- Accessing the Cluster and running kubectl commands: 

Error from server (Forbidden): nodes is forbidden: User "e4d1e37d-ca69-45db-b3bd-06ac186e83a9" cannot list resource "nodes" in API group "" at the cluster scope: User does not have access to the resource in Azure. Update role assignment to allow access.

- Tests: Did not work

1- Check RBAC Role Assignment: Verify the RBAC role assignment for your Azure user in the AKS cluster. You need to ensure that the user has the appropriate role (e.g., Azure Kubernetes Service Cluster User) assigned to the AKS resource. You can check the role assignment using the following command:

az role assignment list --resource-group --assignee

2- Assign RBAC Role: If the user does not have the necessary role assignment, assign the Azure Kubernetes Service Cluster User role to the user:

az role assignment create --role "Azure Kubernetes Service Cluster User" --assignee --scope

Test: Could not find Azure Kubernetes Service Cluster User so I used Azure Kubernetes Service RBAC Reader as well as Admin after

az role assignment create --role "Azure Kubernetes Service RBAC Reader" --assignee e4d1e37d-ca69-45db-b3bd-06ac186e83a9 --scope /subscriptions/ec07bfd1-5848-4234-a145-9a3e0e96d890/resourceGroups/azrez/providers/Microsoft.Kubernetes/connectedClusters/arc-aks-3042

az role assignment create --role "Azure Kubernetes Service RBAC Cluster Admin" --assignee e4d1e37d-ca69-45db-b3bd-06ac186e83a9 --scope /subscriptions/ec07bfd1-5848-4234-a145-9a3e0e96d890/resourceGroups/azrez/providers/Microsoft.Kubernetes/connectedClusters/arc-aks-3042

Added roles to my RG + K8S Service Tried also - https://learn.microsoft.com/en-us/azure/aks/kubernetes-portal?tabs=azure-cli#troubleshooting

Still same problem:

Error from server (Forbidden): deployments.apps is forbidden: User "e4d1e37d-ca69-45db-b3bd-06ac186e83a9" cannot list resource "deployments" in API group "apps" at the cluster scope: User does not have access to the resource in Azure. Update role assignment to allow access.

marianleica commented 2 days ago

Issue didn't reproduce on my side

Error is due to Kubernetes RBAC not disabled

Error from server (Forbidden): deployments.apps is forbidden: User "e4d1e37d-ca69-45db-b3bd-06ac186e83a9" cannot list resource "deployments" in API group "apps" at the cluster scope: User does not have access to the resource in Azure. Update role assignment to allow access.

Please run once more, reproduce the error, next apply the following manifest with your Azure user instead of azure-user@domain.com, and see if you have the access:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: user-full-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: user-full-access
subjects:
- kind: User
  name: <azure-user@domain.com>

If it works, we can automate in the deployments. Generally, I would not prefer to disable local accounts due to various Kubernetes scenarios users might want to do

Andu1391 commented 2 days ago

PS C:\Temp> kubectl apply -f .\Manifest.yml Error from server (Forbidden): error when retrieving current configuration of: Resource: "rbac.authorization.k8s.io/v1, Resource=rolebindings", GroupVersionKind: "rbac.authorization.k8s.io/v1, Kind=RoleBinding" Name: "user-full-access", Namespace: "default" from server for: ".\Manifest.yml": rolebindings.rbac.authorization.k8s.io "user-full-access" is forbidden: User "e4d1e37d-ca69-45db-b3bd-06ac186e83a9" cannot get resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default": User does not have access to the resource in Azure. Update role assignment to allow access.

marianleica commented 2 days ago

Strange, testing.