kubernetes / ingress-nginx

Ingress-NGINX Controller for Kubernetes
https://kubernetes.github.io/ingress-nginx/
Apache License 2.0
17.26k stars 8.2k forks source link

ingress nginx installation issue for livenessProbe and readinessProbe #9335

Closed vikrantoct7 closed 1 year ago

vikrantoct7 commented 1 year ago

We are doing provisioning for Azure kubernetes and trying to deploy nginx ingress using helm. Below is helm command. We downloaded helmchart and added in code repo. we are trying to deploy helm chart using folder location REGISTRY_NAME=$(REGISTRY_NAME) CONTROLLER_IMAGE=ingress-nginx/controller CONTROLLER_TAG=v1.5.1 PATCH_IMAGE=ingress-nginx/kube-webhook-certgen PATCH_TAG=v20220916-gd32f8c343 DEFAULTBACKEND_IMAGE=defaultbackend-amd64 DEFAULTBACKEND_TAG=1.5 ACR_URL=$(REGISTRY_NAME).azurecr.io helm install --replace ingress-nginx $(System.DefaultWorkingDirectory)/ingress-nginx \ --namespace ingress-nginx \ --create-namespace \ --set controller.replicaCount=2 \ --set controller.nodeSelector."kubernetes.io/os"=linux \ --set controller.image.registry=$ACR_URL \ --set controller.image.image=$CONTROLLER_IMAGE \ --set controller.image.tag=$CONTROLLER_TAG \ --set controller.image.digest="" \ --set controller.admissionWebhooks.patch.nodeSelector."kubernetes.io/os"=linux \ --set controller.service.annotations."service.beta.kubernetes.io/azure-load-balancer-health-probe-request-path"=/healthz \ --set controller.admissionWebhooks.patch.image.registry=$ACR_URL \ --set controller.admissionWebhooks.patch.image.image=$PATCH_IMAGE \ --set controller.admissionWebhooks.patch.image.tag=$PATCH_TAG \ --set controller.admissionWebhooks.patch.image.digest="" \ --set defaultBackend.nodeSelector."kubernetes.io/os"=linux \ --set defaultBackend.image.registry=$ACR_URL \ --set defaultBackend.image.image=$DEFAULTBACKEND_IMAGE \ --set defaultBackend.image.tag=$DEFAULTBACKEND_TAG \ --set defaultBackend.image.digest="" \ -f $(System.DefaultWorkingDirectory)/internal-ingress.yaml Followed MS article https://learn.microsoft.com/en-us/azure/aks/ingress-basic?tabs=azure-cli

Customer has enabled a Azure policy "Ensure cluster containers have readiness or liveness probes configured" at subscription level. Due to this policy, we are getting below error

Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev2containerenforceprob-2beef1c35b38e59820d1] Container in your Pod has no . Required probes: ["readinessProbe", "livenessProbe"]

Not sure what and where we are doing wrong. Why this policy is violating however when i do deployment using yaml base https://github.com/kubernetes/ingress-nginx/blob/main/deploy/static/provider/cloud/deploy.yaml then it successful with making bit changes. guide us to resolve this issue

k8s-ci-robot commented 1 year ago

@vikrantoct7: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
longwuyuan commented 1 year ago

Project recommends this https://kubernetes.github.io/ingress-nginx/deploy/#azure

longwuyuan commented 1 year ago

Does that documented method successfully create a run the ingress-nginx-controller for you ?

vikrantoct7 commented 1 year ago

No... our AKS is in private secure env. kubeclt is not recommended solution. Helm terraform is only a solution to deploy helm chart we are using terraform helm release module (https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release)

it seems that when expect node name to pass like set controller.admissionWebhooks.patch.nodeSelector."kubernetes.io/os" (use double quotes) but when we are using in terraform module (no double quotes in name) set { name = "controller.nodeSelector.kubernetes\.io/os" value = "linux" }

we tried to use but this also did does not work set { name = "controller.nodeSelector.\"kubernetes\.io/os\"" value = "linux" }

looking your suggestion

longwuyuan commented 1 year ago
vikrantoct7 commented 1 year ago

Hi team

it seems that there is really a issue with ingress-nginx helm chart.. i wanted to attached full log but could not attach here when i do helm ingress-ngix install then it does not get install and through through below warrning(policy is enabled so this warning comes)..when i delete ingress-nginx namespace and again install, then it gets install successfully..if i again delete ingress-nginx namespace and install again then again i see below warning. not sure why it is happening. Given trace snaps after this warning...please let me know email id, i will share full log

"Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev2containerenforceprob-f438eeab337a1881a882] Container in your Pod has no . Required probes: ["readinessProbe", "livenessProbe"] [azurepolicy-k8sazurev2containerenforceprob-f438eeab337a1881a882] Container in your Pod has no . Required probes: ["readinessProbe", "livenessProbe"]"

snaps of logs is here 2022-11-30T13:51:53.464Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:53 [DEBUG] [resourceReleaseCreate: ingress-nginx] Installing chart: timestamp=2022-11-30T13:51:53.464Z 2022-11-30T13:51:54.226Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:54.226Z 2022-11-30T13:51:54.383Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] Starting delete for "ingress-nginx-admission" ServiceAccount: timestamp=2022-11-30T13:51:54.383Z 2022-11-30T13:51:54.391Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] serviceaccounts "ingress-nginx-admission" not found: timestamp=2022-11-30T13:51:54.391Z 2022-11-30T13:51:54.512Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:54.512Z 2022-11-30T13:51:54.546Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] Starting delete for "ingress-nginx-admission" ClusterRole: timestamp=2022-11-30T13:51:54.546Z 2022-11-30T13:51:54.553Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] clusterroles.rbac.authorization.k8s.io "ingress-nginx-admission" not found: timestamp=2022-11-30T13:51:54.553Z 2022-11-30T13:51:54.691Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:54.691Z 2022-11-30T13:51:54.724Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] Starting delete for "ingress-nginx-admission" ClusterRoleBinding: timestamp=2022-11-30T13:51:54.724Z 2022-11-30T13:51:54.731Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] clusterrolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" not found: timestamp=2022-11-30T13:51:54.731Z 2022-11-30T13:51:54.871Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:54.871Z 2022-11-30T13:51:54.912Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] Starting delete for "ingress-nginx-admission" Role: timestamp=2022-11-30T13:51:54.911Z 2022-11-30T13:51:54.919Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:54 [DEBUG] roles.rbac.authorization.k8s.io "ingress-nginx-admission" not found: timestamp=2022-11-30T13:51:54.919Z 2022-11-30T13:51:55.060Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:55.060Z 2022-11-30T13:51:55.093Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] Starting delete for "ingress-nginx-admission" RoleBinding: timestamp=2022-11-30T13:51:55.093Z 2022-11-30T13:51:55.100Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] rolebindings.rbac.authorization.k8s.io "ingress-nginx-admission" not found: timestamp=2022-11-30T13:51:55.100Z 2022-11-30T13:51:55.232Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:55.232Z 2022-11-30T13:51:55.265Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] Starting delete for "ingress-nginx-admission-create" Job: timestamp=2022-11-30T13:51:55.265Z 2022-11-30T13:51:55.273Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] jobs.batch "ingress-nginx-admission-create" not found: timestamp=2022-11-30T13:51:55.272Z 2022-11-30T13:51:55.395Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] creating 1 resource(s): timestamp=2022-11-30T13:51:55.395Z 2022-11-30T13:51:55.415Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] Watching for changes to Job ingress-nginx-admission-create with timeout of 5m0s: timestamp=2022-11-30T13:51:55.415Z 2022-11-30T13:51:55.422Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] Add/Modify event for ingress-nginx-admission-create: ADDED: timestamp=2022-11-30T13:51:55.422Z 2022-11-30T13:51:55.422Z [INFO] provider.terraform-provider-helm_v2.7.1_x5: 2022/11/30 13:51:55 [DEBUG] ingress-nginx-admission-create: Jobs active: 0, jobs failed: 0, jobs succeeded: 0: timestamp=2022-11-30T13:51:55.422Z 2022-11-30T13:51:58.280Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" 2022-11-30T13:51:58.280Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" is waiting for "helm_release.ingress-nginx" 2022-11-30T13:52:03.281Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" is waiting for "helm_release.ingress-nginx" 2022-11-30T13:52:03.281Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" helm_release.ingress-nginx: Still creating... [10s elapsed] 2022-11-30T13:52:08.283Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" 2022-11-30T13:52:08.283Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" is waiting for "helm_release.ingress-nginx" 2022-11-30T13:52:13.285Z [TRACE] dag/walk: vertex "provider[\"registry.terraform.io/hashicorp/helm\"] (close)" is waiting for "helm_release.ingress-nginx" 2022-11-30T13:52:13.285Z [TRACE] dag/walk: vertex "root" is waiting for "provider[\"registry.terraform.io/hashicorp/helm\"] (close)"

tao12345666333 commented 1 year ago

"Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [azurepolicy-k8sazurev2containerenforceprob-f438eeab337a1881a882] Container in your Pod has no . Required probes: ["readinessProbe", "livenessProbe"] [azurepolicy-k8sazurev2containerenforceprob-f438eeab337a1881a882] Container in your Pod has no . Required probes: ["readinessProbe", "livenessProbe"]"

This indicates that your policy engine expects the configuration to be included in the deployment manifest.

tao12345666333 commented 1 year ago

And you can check your policy named azurepolicy-k8sazurev2containerenforceprob-f438eeab337a1881a882

strongjz commented 1 year ago

/kind support /triage needs-information

vikrantoct7 commented 1 year ago

It seems couple of pods are not having readiness and liveness probes due to that it was failing. I understand that those pods are not web type not it is not worth to have readiness and liveness probes... Not sure to whom we can approach to fix.. as an workaround we modified the helm chart to solve our problem.

strongjz commented 1 year ago

@vikrantoct7 if you'd like to open a PR to let us see your resolution to the issue that would be helpful for others.

For now, I'm closing this issue since you have a workaround

/close

k8s-ci-robot commented 1 year ago

@strongjz: Closing this issue.

In response to [this](https://github.com/kubernetes/ingress-nginx/issues/9335#issuecomment-1472991022): >@vikrantoct7 if you'd like to open a PR to let us see your resolution to the issue that would be helpful for others. > >For now, I'm closing this issue since you have a workaround > >/close > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.