Closed flyer299 closed 8 months ago
@flyer299 You need to make a minor change to your ingress:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: voteapp
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- host: vote.dev.speakwrite.app
- http: # remove the the dash before http
paths:
- backend:
serviceName: azure-vote-front
servicePort: 80
Please remove the dash before http. A dash means a new item in the list. So the host and http in your are considered two separate rules instead of one. You can use this tutorial for reference.
Thanks for pointing. Here is an example how you can do path-based routing.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: voteapp
annotations:
kubernetes.io/ingress.class: azure/application-gateway
spec:
rules:
- host: vote.dev.speakwrite.app
http:
paths:
- path: /app-a
backend:
serviceName: app-a
servicePort: 80
- path: /app-b
backend:
serviceName: app-b
servicePort: 80
That helped me move forward a bit... But now I am having a problem once I've added LetsEncrypt to the mix.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: guestbook-letsencrypt-staging
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-prod
cert-manager.io/acme-challenge-type: http01
spec:
tls:
- hosts:
- guestbook.dev.speakwrite.app
secretName: guestbook-secret-name
rules:
- host: guestbook.dev.speakwrite.app
http:
paths:
- backend:
serviceName: frontend
servicePort: 80
The AGIC Pod now gives this error:
Event(v1.ObjectReference{Kind:"Ingress", Namespace:"default", Name:"guestbook-letsencrypt-staging", UID:"7f53c972-cc12-4b42-a662-5293c3c3ad00", APIVersion:"extensions/v1beta1", ResourceVersion:"3024037", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [default/guestbook-secret-name]
I saw this issue: https://github.com/Azure/application-gateway-kubernetes-ingress/issues/721 which is the same error I am getting. But the resolution to that issue has already been applied in my examples.
What is causing the SSL to not work?
Here is my ClusterIssuer
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-prod
spec:
acme:
# You must replace this email address with your own.
# Let's Encrypt will use this to contact you about expiring
# certificates, and issues related to your account.
email: itaccounts@speakwrite.com
# ACME server URL for Let’s Encrypt’s staging environment.
# The staging environment will not issue trusted certificates but is
# used to ensure that the verification process is working properly
# before moving to production
server: https://acme-v01.api.letsencrypt.org/directory
privateKeySecretRef:
# Secret resource used to store the account's private key.
name: letsencrypt-secret
# Enable the HTTP-01 challenge provider
# you prove ownership of a domain by ensuring that a particular
# file is present at the domain
solvers:
- http01:
ingress:
class: azure/application-gateway
I have these items running in the cert-manager namespace:
kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$ kubectl get services --namespace cert-manager
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cert-manager ClusterIP 10.2.33.11 <none> 9402/TCP 85m
cert-manager-webhook ClusterIP 10.2.97.176 <none> 443/TCP 85m
kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$ kubectl get deployments --namespace cert-manager
NAME READY UP-TO-DATE AVAILABLE AGE
cert-manager 1/1 1 1 86m
cert-manager-cainjector 1/1 1 1 86m
cert-manager-webhook 1/1 1 1 86m
kveard@DEV-LT01:/mnt/c/Users/ken.veard/source/AKS/AGIC/LetsEncrypt$
@flyer299 Were you able to get past this issue ? Did you check if the secret exists ?
I was never able to get Let’s Encrypt working with Application Gateway. I had to revert back to an NGIX Ingress Controller. I am going to wait a month or two and check back on Application Gateway. I really want App Gateway for our solution when we go to production, but I want to give the teams behind it more time to make the integration with Government cloud more robust and straight forward before I invest any more of our time on it.
From: Akshay Gupta notifications@github.com Sent: Friday, April 3, 2020 7:56 PM To: Azure/application-gateway-kubernetes-ingress application-gateway-kubernetes-ingress@noreply.github.com Cc: Ken VeArd ken@veard.net; Mention mention@noreply.github.com Subject: Re: [Azure/application-gateway-kubernetes-ingress] Comprehensive Reference/Documentation (#758)
@flyer299https://github.com/flyer299 Were you able to get past this issue ? Did you check if the secret exists ?
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/Azure/application-gateway-kubernetes-ingress/issues/758#issuecomment-608934691, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABURFKGJ7YMYTPYOMUZYEE3RK2AR5ANCNFSM4LDF7XFA.
I am facing the same issue.
I am trying to install LetsEncrypt via helm into a terraform deployed cluster using gateway. Here are the relevant bits of the terraform code:
resource "kubernetes_namespace" "agic_ns" {
metadata {
name = "agic"
}
}
resource "helm_release" "agic" {
name = "agic"
repository = "https://appgwingress.blob.core.windows.net/ingress-azure-helm-package"
chart = "ingress-azure"
namespace = kubernetes_namespace.agic_ns.metadata.0.name
version = "1.2.0"
set {
name = "verbosityLevel"
value = 5
}
set {
... lots of irrelevant stuff here ...
}
}
resource "kubernetes_namespace" "cert_ns" {
metadata {
name = "cert-manager"
}
}
resource "helm_release" "cert_manager" {
repository = "https://charts.jetstack.io"
name = "cert-manager"
chart = "cert-manager"
namespace =kubernetes_namespace.cert_ns.metadata.0.name
version = "v1.0.2"
set {
name = "installCRDs"
value = true
}
}
Firstly, I try to deploy the ingress with a static passwordless pks cert in a keyvault (pod identity helm used here). THIS WORKS!
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
namespace: my-svc-ns
annotations:
kubernetes.io/ingress.class: azure/application-gateway
appgw.ingress.kubernetes.io/appgw-ssl-certificate: my-cert
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: www.example.com
http:
paths:
... paths here ...
Next, I make some minor changes in line with the documentation found here: https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/docs/how-tos/lets-encrypt.md
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress
namespace: my-svc-ns
annotations:
kubernetes.io/ingress.class: azure/application-gateway
cert-manager.io/cluster-issuer: letsencrypt-cert
cert-manager.io/acme-challenge-type: http01
appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
tls:
- hosts:
- www.example.com
secretName: letsencrypt-secret
rules:
- host: www.example.com
http:
paths:
... paths here ...
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
name: letsencrypt-cert
namespace: my-svc-ns
spec:
acme:
email: email@example.com
server: https://acme-staging-v02.api.letsencrypt.org/directory # testing certs
# server: https://acme-v02.api.letsencrypt.org/directory # prod certs
privateKeySecretRef:
name: letsencrypt-secret
solvers:
- http01:
ingress:
class: azure/application-gateway
Annoyingly this breaks the app gateway. If I hit http://my-site, I get a 307 and pointed at https://my-site which returns a cannot be reached (not an application-gateway 502 or 404 page!) Inspecting the app gateway layer I get the following error:
frontend_listeners.go:121] Processing Rules for Ingress: my-svc-ns/ingress
Event(v1.ObjectReference{Kind:"Ingress", Namespace:"my-svc-ns", Name:"ingress", UID:"c4cc4899-1dc4-4a23-83d1-da95e54b280c", APIVersion:"extensions/v1beta1", ResourceVersion:"3170126", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: ][my-svc-ns/letsencrypt-secret]
I can inspect the issuer via kubectl describe clusterissuer -A
and it seems to have generated me a (something?) certificate:
Everything looks OK here.
API Version: cert-manager.io/v1
Kind: ClusterIssuer
... stuff here ...
Spec:
Acme:
Email: example@example.com
Preferred Chain:
Private Key Secret Ref:
Name: letsencrypt-secret
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Solvers:
http01:
Ingress:
Class: azure/application-gateway
Status:
Acme:
Last Registered Email: example@example.com
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/16112602
Conditions:
Last Transition Time: 2020-10-14T12:18:34Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
I can inspect the secrets via: kubectl get secrets -n cert-manager
and I get the following:
cert-manager cert-manager-cainjector-token-ms887 kubernetes.io/service-account-token 3 20h
cert-manager cert-manager-token-gr2rt kubernetes.io/service-account-token 3 20h
cert-manager cert-manager-webhook-ca Opaque 3 20h
cert-manager cert-manager-webhook-token-lgd4n kubernetes.io/service-account-token 3 20h
cert-manager default-token-cdtws kubernetes.io/service-account-token 3 20h
cert-manager letsencrypt-secret Opaque 1 19h
cert-manager sh.helm.release.v1.cert-manager.v1 helm.sh/release.v1 1 20h
Note that via kubectl describe secret -n cert-manager letsencrypt-secret
I can see that a .key file has been created:
Name: letsencrypt-secret
Namespace: cert-manager
Type: Opaque
Data
====
tls.key: 1675 bytes
Next up, as per this Microsoft documentation: https://docs.microsoft.com/en-us/azure/aks/ingress-static-ip
I inspect the certificates via kubectl get certificates -A
NAMESPACE NAME READY SECRET AGE
my-svc-ns letsencrypt-secret False letsencrypt-secret 17h
Now it looks like its an issue with the certificate, not the app gateway.
I jump over to this documentation: https://cert-manager.io/docs/faq/troubleshooting/
and run kubectl describe certificate letsencrypt-secret -n my-svc-ns
Name: letsencrypt-secret
Namespace: my-svc-ns
... boring stuff here ...
Status:
Message: The certificate request has failed to complete and will be retried: Failed to wait for order resource "letsencrypt-secret-cxg5p-2149327961" to become ready: order is in "invalid" state:
Reason: Failed
Status: False
Type: Issuing
Message: Issuing certificate as Secret does not exist
Reason: DoesNotExist
Status: False
Type: Ready
Last Failure Time: 2020-10-15T07:01:05Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 56m (x18 over 17h) cert-manager Issuing certificate as Secret does not exist
Normal Generated 56m cert-manager Stored new private key in temporary Secret resource "letsencrypt-secret-wcjnp"
Warning Failed 56m cert-manager The certificate request has failed to complete and will be retried: Failed to wait for order resource "letsencrypt-secret-cxg5p-2149327961" to become ready: order is in "invalid" state:
Normal Requested 56m cert-manager Created new CertificateRequest resource "letsencrypt-secret-lrzgf"
Next up, it looks like the order has failed kubectl describe certificaterequest letsencrypt-secret-x8csq -n my-svc-ns
Name: letsencrypt-secret-x8csq
Namespace: my-svc-ns
... more stuff here ...
Status:
Conditions:
Last Transition Time: 2020-10-15T08:01:05Z
Message: Failed to wait for order resource "letsencrypt-secret-x8csq-2149327961" to become ready: order is in "invalid" state:
Reason: Failed
Status: False
Type: Ready
Failure Time: 2020-10-15T08:02:02Z
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal OrderCreated 5m30s cert-manager Created Order resource my-svc-ns/letsencrypt-secret-x8csq-2149327961
Warning OrderFailed 4m33s cert-manager Failed to wait for order resource "letsencrypt-secret-x8csq-2149327961" to become ready: order is in "invalid" state:
Again, you can check the order via kubectl describe orders -A
Name: letsencrypt-secret-x8csq-2149327961
Namespace: my-svc-ns
... more of the boring stuff ...
Status:
Authorizations:
Challenges:
Token: <REDACTED>
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
Token: <REDACTED>
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
Token: <REDACTED>
Type: tls-alpn-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/<REDACTED>
Identifier: example.com
Initial State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/<REDACTED>
Wildcard: false
Failure Time: 2020-10-15T08:02:02Z
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/<REDACTED>
State: invalid
URL: https://acme-v02.api.letsencrypt.org/acme/order/<REDACTED>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Created 10m cert-manager Created Challenge resource "letsencrypt-secret-x8csq-<redacted>" for domain "example.com"
Looks like the challenge is in an invalid state. Down the rabbit hole we go!! I looked through the service logs and found that the challenge is responding with some html, rather than the actual response.
Here I am using stern to extract the service logs in one go:
stern -n cert-manager cert-manager-webhook -s 2h >> /c/Downloads/cert-manager-webhook-logs.txt
stern -n cert-manager cert-manager -s 2h >> /c/Downloads/cert-manager-logs.txt
I have gone though the logs. Here the the relevant bits:
cert-manager-webhook logs
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.598962 1 logs.go:58] http: TLS handshake error from 10.2.0.39:56436: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.601111 1 logs.go:58] http: TLS handshake error from 10.2.0.39:56438: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 12:01:05.613115 1 logs.go:58] http: TLS handshake error from 10.2.0.39:56442: EOF
cert-manager-webhook-7b56df6ddb-bfhx7 cert-manager I1015 13:01:05.614956 1 logs.go:58] http: TLS handshake error from 10.2.0.39:33458: EOF
cert-manager logs
12:01:05.598962 1 logs.go:58] http: TLS handshake error from 10.2.0.39:56436: EOF
... this message repeated ~ 5 times ...
11:30:51.601008 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" ... variations on this message for a few hundred lines ... 11:30:51.601036 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled"
12:01:05.000769 1 conditions.go:162] Found status change for Certificate "letsencrypt-secret" condition "Issuing": "False" -> "True"; setting lastTransitionTime to 2020-10-15 12:01:05.000756269 +0000 UTC m=+7349.678490276 13:19:32.480187 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" "controller"="apiservice" "name"="v1beta1.external.metrics.k8s.io" "namespace"="" "reconcilerGroup"="apiregistration.k8s.io" "reconcilerKind"="APIService" ... variations on this message for a few dozen lines ... 13:19:32.480368 1 controller.go:261] cert-manager/controller "msg"="Successfully Reconciled" 12:01:05.456313 1 conditions.go:162] Found status change for Certificate "letsencrypt-secret" condition "Issuing": "True" -> "False";
12:01:05.470695 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-acme "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474184 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-ca "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474233 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-selfsigned "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474254 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-vault "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.474313 1 controller.go:184] cert-manager/controller/certificaterequests-issuer-venafi "msg"="certificate request in work queue no longer exists" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found"
12:01:05.477599 1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="my-svc-ns/letsencrypt-secret" "retry_after"="2020-10-15T13:01:05Z" 12:01:05.488750 1 util.go:71] cert-manager/controller/certificaterequests/handleOwnedResource "msg"="error getting referenced owning resource" "error"="certificaterequest.cert-manager.io \"letsencrypt-secret-zk5q5\" not found" "related_resource_kind"="CertificateRequest" 12:01:05.488872 1 controller.go:142] cert-manager/controller/orders "msg"="order in work queue no longer exists" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found"
12:01:05.519805 1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found" 12:01:05.589204 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"letsencrypt-secret\": the object has been modified; please apply your changes to the latest version and try again" "key"="my-svc-ns/letsencrypt-secret"
12:01:05.621883 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "letsencrypt-secret-dbgxb" condition "Ready" to 2020-10-15 12:01:05.621874877 +0000 UTC m=+7350.299608884 12:01:05.695702 1 util.go:71] cert-manager/controller/orders/handleOwnedResource "msg"="error getting referenced owning resource" "error"="order.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961\" not found" 12:01:05.696288 1 controller.go:196] cert-manager/controller/challenges "msg"="challenge in work queue no longer exists" "error"="challenge.acme.cert-manager.io \"letsencrypt-secret-zk5q5-2149327961-1861071662\" not found"
12:01:09.892470 1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod"
12:01:09.934564 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod"
12:01:09.935267 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource"
12:01:09.935640 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress"
12:01:10.023099 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="did not get expected response when querying endpoint, expected \"
12:02:00.229824 1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" 12:02:00.229930 1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" 12:02:00.230006 1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" 12:02:00.242971 1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '502', expected '200'" ... this message repeated ~ 10 times ...
12:03:02.337775 1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for example.com: 403 urn:ietf:params:acme:error:caa: CAA record for example.com prevents issuance"
The following error stands out:
acme: authorization error for example.com: 403 urn:ietf:params:acme:error:caa: CAA record for example.com prevents issuance
But this is likely due to the previous 502 and error pages:
wrong status code '502', expected '200'
did not get expected response when querying endpoint, expected \"
Frustratingly, the HTML is truncated so I am unable to fully see if it is my webapp (served via ingress route /*) or if its the good old fashioned 404 / 502 page that app gateway produces.
Something fishy is going on though, I followed the azure documentation almost exactly and it seems like the gateway itself is passing back a 502.
==================
I appreciate that the cert-manager app isn't an Azure product, but getting an SSL cert for your ingress is a vital step that most people using the gateway will need to do!
It seems to me that some extra step is missing from the instructions here. Or maybe my configuration is wrong?
@akshaysngupta ?
@thatguycalledrob thanks a lot for such a detailed post. Let me reproduce this and post back.
Facing a similar issue. This is a major blocker to adoption.
@akshaysngupta - Wanted to give you an update.
My problem was incorrectly set CAA records. To sys admins, proper setup of the CAA record should be an obvious step, but in my case as a backend-dev-doing-devops it wasn't.
For my organisation our CAA records are set via ticket to a central DNS team (sigh), and its a whitelist with no default values. After reading the cert-bot docs for another project, I realised that I hadn't checked the CAA record. One IT ticket later and my above issue went away!
Since the error logs are pretty rubbish in this case, it may be worth adding some comment around CAA record setup for the uninitiated!
Facing the same issue here. May I know anyone got a resolution already? or it's still the same, like a bug?
Is there a place that I am not seeing that will give me comprehensive documentation & reference for the different types of Ingress Options that are available with AGIC in the Ingress Yaml file? There are several use cases I want to start exploring, but I feel like a dummy for not even knowing where to start.
They include:
I have tried the first bullet point and I can't get it to work. I can only access the first site I deploy via IP address. Even though I set the ingress up with a host name, I don't think the Gateway is getting correctly configured by the AGIC. But I am struggling on where to troubleshoot and what to do differently.
My App's Deployment:
My Ingress:
Logs from AGIC Pod