Closed vduplessis-stonal closed 1 year ago
M00nF1sh
commented
1 year ago @vduplessis-stonal
do you have any controllers that deleted the Ingresses immediately?
from the logs, {"level":"info","ts":"2023-04-26T14:25:53Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"whoami/whoami\",\"resources\":{}}"}, it seems the Ingress were deleted.
Would you help check the kube-apiserver audit log to confirm that?
vduplessis-stonal
commented
1 year ago Hello @M00nF1sh, Thank you for your response. The Ingress is not deleted at any time during the process, in fact it even have this after the initial creation:
status:
loadBalancer:
ingress:
- hostname: k8s-whoami-whoami-ecdc81edf9-286150540.eu-west-3.elb.amazonaws.comThe hostname here is the address of an ALB that does not exist in the AWS console, which indicates for me that there was indeed an ALB at some point that was immediately deleted. The only event it has is this one (the age corresponds to the moment I created the Ingress):
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyReconciled 26m ingress Successfully reconciledHello,
We have found something troubling while investigating the kube-apiserver audit logs. It seems the Load Balancer Controller modify the ingress right after the creation. We found this log:
{
"kind": "Event",
"apiVersion": "audit.k8s.io/v1",
"level": "RequestResponse",
"auditID": "96014ec0-79f5-4cbc-b34f-37bc5339eacf",
"stage": "ResponseComplete",
"requestURI": "/apis/networking.k8s.io/v1/namespaces/whoami/ingresses/whoami",
"verb": "patch",
"user": {
"username": "system:serviceaccount:kube-system:aws-load-balancer-controller",
"uid": "e2a36511-c62b-4681-8a25-deff13acccc2",
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:kube-system",
"system:authenticated"
],
"extra": {
"authentication.kubernetes.io/pod-name": [
"aws-load-balancer-controller-85649d7dd5-s8w5p"
],
"authentication.kubernetes.io/pod-uid": [
"fcbf3337-56b5-4c13-8551-859810f4c4a8"
]
}
},
[...]
"responseObject": {
"kind": "Ingress",
"apiVersion": "networking.k8s.io/v1",
"metadata": {
"name": "whoami",
"namespace": "whoami",
"uid": "f905611d-4220-422b-b006-8f7a05507427",
"resourceVersion": "33765621",
"generation": 2,
"creationTimestamp": "2023-05-03T08:50:44Z",
"annotations": {
"alb.ingress.kubernetes.io/certificate-arn": "arn:aws:acm:eu-west-3:983974232060:certificate/8d5074d5-9e01-4722-8503-f9834e7103c4",
"alb.ingress.kubernetes.io/listen-ports": "[{\"HTTPS\":443}]",
"alb.ingress.kubernetes.io/scheme": "internet-facing",
"alb.ingress.kubernetes.io/target-type": "ip",
"external-dns.alpha.kubernetes.io/hostname": "*.test.eks-sandbox.stonal.io",
"field.cattle.io/publicEndpoints": "[{\"addresses\":[\"\"],\"port\":443,\"protocol\":\"HTTPS\",\"serviceName\":\"whoami:whoami\",\"ingressName\":\"whoami:whoami\",\"hostname\":\"whoami.test.eks-sandbox.stonal.io\",\"path\":\"/\",\"allNodes\":false}]",
"kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"networking.k8s.io/v1\",\"kind\":\"Ingress\",\"metadata\":{\"annotations\":{\"alb.ingress.kubernetes.io/certificate-arn\":\"arn:aws:acm:eu-west-3:983974232060:certificate/8d5074d5-9e01-4722-8503-f9834e7103c4\",\"alb.ingress.kubernetes.io/listen-ports\":\"[{\\\"HTTPS\\\":443}]\",\"alb.ingress.kubernetes.io/scheme\":\"internet-facing\",\"alb.ingress.kubernetes.io/target-type\":\"ip\",\"external-dns.alpha.kubernetes.io/hostname\":\"*.test.eks-sandbox.stonal.io\"},\"name\":\"whoami\",\"namespace\":\"whoami\"},\"spec\":{\"ingressClassName\":\"alb\",\"rules\":[{\"host\":\"whoami.test.eks-sandbox.stonal.io\",\"http\":{\"paths\":[{\"backend\":{\"service\":{\"name\":\"whoami\",\"port\":{\"number\":80}}},\"path\":\"/\",\"pathType\":\"Prefix\"}]}}],\"tls\":[{\"hosts\":[\"whoami.test.eks-sandbox.stonal.io\"]}]}}\n"
},
[...]
"spec": {
"tls": [
{
"hosts": [
"whoami.test.eks-sandbox.stonal.io"
]
}
],
"rules": [
{
"host": "whoami.test.eks-sandbox.stonal.io",
"http": {
"paths": [
{
"path": "/",
"pathType": "ImplementationSpecific",
"backend": {
"service": {
"name": "whoami",
"port": {
"number": 80
}
}
}
}
]
}
}
]
},
"status": {
"loadBalancer": {
"ingress": [
{
"hostname": "k8s-whoami-whoami-ecdc81edf9-1489757877.eu-west-3.elb.amazonaws.com"
}
]
}
}
},As it shows here, it seems the patch operation removed the ingressClassName: alb field from the specs, and it also changed the pathType. Is this normal behaviour from the controller ?
oliviassss
commented
1 year ago @vduplessis-stonal, I don't think the controller should remove the ingressClassName or change the pathType. do you have other controllers installed that delete/modify the resources created by the AWS LBC? or do you have multiple AWS LB controllers?
oliviassss
commented
1 year ago @vduplessis-stonal, I'm going to close the issue as for now, please feel free to reach out or reopen if you have any questions, thanks.
Describe the bug I am currently trying to migrate from nginx ingress controller to LB ingress controller with ALB. I am trying to deploy a simple whoami service to test the ingress, which is as follow:
When applying this, I get this result on the controller logs:
To summarize, the AWS resources (Load Balancer, Target Group, Security Group) are immediately deleted right after being created. When I reapply the exact same configuration after that, it creates it normally.
Expected outcome The ALB and the rest of the other AWS resources should not be deleted right after being created when adding a new Ingress.
Environment