search

Azure / application-gateway-kubernetes-ingress

This is an ingress controller that can be run on Azure Kubernetes Service (AKS) to allow an Azure Application Gateway to act as the ingress for an AKS cluster.
https://azure.github.io/application-gateway-kubernetes-ingress
MIT License
667 stars 413 forks source link

AGIC is cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope #1582

Closed AruVenkat closed 4 months ago

AruVenkat commented 7 months ago

Describe the bug I'm using AGIC in my AKS cluster. once, I removed one of my ingress and the changes weren't reflected in the Application Gateway. the backend pool wasn't removed and I'm getting a 502 gateway error. when I saw the logs, I found the below error continuously,

E1121 22:51:59.744258 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: azureapplicationgatewayrewrites.appgw.ingress.azure.io is forbidden: User "system:serviceaccount:kube-system:ingress-appgw-sa" cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope I1121 22:52:00.796911 1 reflector.go:530] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Watch close - v1.Secret total 46 items received I1121 22:52:19.582654 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:19.582682 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:19.582738 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:19.582750 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:19.582741 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:19.803122 1 reflector.go:381] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: forcing resync I1121 22:52:46.881629 1 reflector.go:255] Listing and watching v1beta1.AzureApplicationGatewayRewrite from pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167

any assistance would be appreciated :) Thanks in advance

akshaysngupta commented 7 months ago

AGIC's role binding has appgw.ingress.azure.io. What version are you running ? https://github.com/Azure/application-gateway-kubernetes-ingress/blob/a11719a9f0d5e95edad0e67c9d81a6486a3f1d93/helm/ingress-azure/templates/clusterrole.yaml#L29C7-L29C31

akshaysngupta commented 7 months ago

Also, are you using addon ?

Standy0215 commented 7 months ago

Same issue found for my environment.

I'm using AKS addon and the cluster is upgraded from 1.25 to 1.27 for using AGIC 1.7.2.

Then, the AGIC pod starts reporting issue that it can't watch the azureapplicationgatewayrewrites.appgw.ingress.azure.io.

For my cluster role used by the AGIC pod, it has the permission to .appgw.ingress.k8s.io. Seems that ".appgw.ingress.azure.io" is missing here.

Here is the result of "kubectl describe clusterrole ingress-appgw-cr". 

Name:         ingress-appgw-cr
Labels:       addonmanager.kubernetes.io/mode=Reconcile
              app=ingress-appgw
Annotations:  <none>
PolicyRule:
  Resources                           Non-Resource URLs  Resource Names  Verbs
  ---------                           -----------------  --------------  -----
  events                              []                 []              [get list watch create patch]
  endpoints                           []                 []              [get list watch]
  namespaces                          []                 []              [get list watch]
  nodes                               []                 []              [get list watch]
  pods                                []                 []              [get list watch]
  secrets                             []                 []              [get list watch]
  services                            []                 []              [get list watch]
  *.appgw.ingress.k8s.io              []                 []              [get list watch]
  ingressclasses.extensions           []                 []              [get list watch]
  ingresses.extensions                []                 []              [get list watch]
  *.networking.istio.io               []                 []              [get list watch]
  ingressclasses.networking.k8s.io    []                 []              [get list watch]
  ingresses.networking.k8s.io         []                 []              [get list watch]
  ingresses.extensions/status         []                 []              [update]
  ingresses.networking.k8s.io/status  []                 []              [update]

The age for my cluster role is quite old here. Seems it's not updated accordingly while the AGIC image is updated.

Besides, the CRD "azureapplicationgatewayrewrites.appgw.ingress.azure.io" seems not installed as I start seeing this error after manually adding the permission in YAML of the cluster role.

E1127 09:28:32.853153 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)

digitalbanana commented 7 months ago

any fix for this? - its a real problem for us... :-(

NathanDunning commented 7 months ago

Same with me, I have deployed a new cluster with Kubernetes version 1.28.3 and AGIC Add-On 1.7.2 and am seeing similar error logs from the AGIC pod:

E1130 02:52:21.528259 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: azureapplicationgatewayrewrites.appgw.ingress.azure.io is forbidden: User "system:serviceaccount:kube-system:ingress-appgw-sa" cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope: Azure does not have opinion for this user.

CJJ69 commented 7 months ago

Same issue for us.

Azure K8S 1.27.7 with AGIC add on 1.7.2

Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: azureapplicationgatewayrewrites.appgw.ingress.azure.io is forbidden: User "system:serviceaccount:kube-system:ingress-appgw-sa" cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope.

ParaTrix commented 7 months ago

Hi, this could be fixed by manually add this API group permission to your service account for AGIC.

agic-roleIssue-tempfix.txt

GersonDias commented 7 months ago

This is a frustrating issue because installing the AGIC with Helm does not cause it. It would also be helpful to have the same options that the Helm installation offers, especially regarding the use of a shared AGW instance.

ParaTrix commented 6 months ago

Hi @akshaysngupta, do we have any eta on fix?

karabasosman commented 6 months ago

@JackStromberg , add-on needs a fix. just fyi.

zioproto commented 6 months ago

@JackStromberg I confirm I can reproduce this bug on AKS v1.27.7

It seems we have this problem since longer. This seems to be a duplicated issue of #1495

kaushikd13 commented 5 months ago

Still seeing these errors. Any solution for this?

E0109 08:27:56.481514 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: azureapplicationgatewayrewrites.appgw.ingress.azure.io is forbidden: User "system:serviceaccount:kube-system:ingress-appgw-sa" cannot list resource "azureapplicationgatewayrewrites" in API group "appgw.ingress.azure.io" at the cluster scope: Azure does not have opinion for this user.

E0109 09:38:27.684411 1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.20.0-beta.1/tools/cache/reflector.go:167: Failed to watch v1beta1.AzureApplicationGatewayRewrite: failed to list v1beta1.AzureApplicationGatewayRewrite: the server could not find the requested resource (get azureapplicationgatewayrewrites.appgw.ingress.azure.io)

hatboyzero commented 5 months ago

I am consistently seeing this issue on four different AKS clusters using AGIC.

dqmicrosoft commented 5 months ago

there are some fixes for terraform or manually on this thread https://github.com/Azure/application-gateway-kubernetes-ingress/issues/1495

akshaysngupta commented 4 months ago

This has been fixed in AGIC addon.