Closed jtcressy closed 7 years ago
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.
means that it cannot reach the route. Are you sure cloud.crl.coloradomesa.edu
is pointing to your http load balancer? Can you share your routes openshift-cloud-console
and letsencrypt-cloud.crl.coloradomesa.edu
(scrub certificates and keys if there are any):
oc get route -o yaml -n default openshift-cloud-console letsencrypt-cloud.crl.coloradomesa.edu
You will have to spin up the letsencrypt dc again of course.
Please also include the output of
oc get endpoints -n default letsencrypt
Also, please try to access http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner
from your browser when openshift-letsencrypt
is trying to get a certificate.
http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge
was giving me a 502 before, but now it's just the generic not-found page generated by the router. So, yes, the connection to the router is working, but i guess something with the router isn't working.
output of oc get route -o yaml -n default openshift-cloud-console letsencrypt-cloud.crl.coloradomesa.edu
:
apiVersion: v1
items:
- apiVersion: v1
kind: Route
metadata:
creationTimestamp: 2017-07-04T09:28:29Z
labels:
butter.sh/letsencrypt-managed: "yes"
name: openshift-cloud-console
namespace: default
resourceVersion: "228513"
selfLink: /oapi/v1/namespaces/default/routes/openshift-cloud-console
uid: 23efc6fc-609b-11e7-94fc-005056857075
spec:
host: cloud.crl.coloradomesa.edu
port:
targetPort: console
tls:
destinationCACertificate: "[redacted]"
termination: reencrypt
to:
kind: Service
name: openshift-cloud-console
weight: 100
wildcardPolicy: None
status:
ingress:
- conditions:
- lastTransitionTime: 2017-07-04T09:28:29Z
status: "True"
type: Admitted
host: cloud.crl.coloradomesa.edu
routerName: router
wildcardPolicy: None
- apiVersion: v1
kind: Route
metadata:
annotations:
openshift.io/generated-by: openshift-letsencrypt
creationTimestamp: 2017-07-04T10:11:28Z
labels:
butter.sh/letsencrypt-domainname: cloud.crl.coloradomesa.edu
butter.sh/letsencrypt-well-defined: "yes"
name: letsencrypt-cloud.crl.coloradomesa.edu
namespace: default
resourceVersion: "231149"
selfLink: /oapi/v1/namespaces/default/routes/letsencrypt-cloud.crl.coloradomesa.edu
uid: 24bb4f8d-60a1-11e7-94fc-005056857075
spec:
host: cloud.crl.coloradomesa.edu
path: /.well-known/acme-challenge
tls:
insecureEdgeTerminationPolicy: Allow
termination: edge
to:
kind: Service
name: letsencrypt
weight: 100
wildcardPolicy: None
status:
ingress:
- conditions:
- lastTransitionTime: 2017-07-04T10:11:28Z
status: "True"
type: Admitted
host: cloud.crl.coloradomesa.edu
routerName: router
wildcardPolicy: None
kind: List
metadata: {}
resourceVersion: ""
selfLink: ""
And output of oc get endpoints -n default letsencrypt
:
NAME ENDPOINTS AGE
letsencrypt 10.131.0.22:8080 59m
Hmm. I have a hunch, it might be the tls-termination "reencrypt" that does not work well with "edge".
I don't quiet understand what error you got at what point. When a backend is not reachable the router will return a 502 (and that not-found-page), I can't think of any reason why that might happen.
When spinning up openshiftt-letsencrypt again, wait for the route to be added (and admitted) and try to access http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner
. Can you check where the request gets routed? Please also check in the access log of openshift-cloud-console
. Because the nginx image does not log accesses, only errors, also try to curl http://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/does-not-exist
. You should see some output in the nginx container of the letsencrypt pod (oc logs letsencrypt-... -c nginx
).
When i go to https://cloud.crl.coloradomesa.edu/.well-known/acme-challenge/.owner
, it does work. it reaches the nginx. Going to just .well-known/acme-challenge
produces a 403 code and i see errors in the nginx log.
Interesting, I got this in the watcher log this time:
watching routes with selector butter.sh/letsencrypt-managed=yes
Processing route /oapi/v1/namespaces/default/routes/openshift-cloud-console with domain cloud.crl.coloradomesa.edu.
unable to load certificate
140526855919520:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for cloud.crl.coloradomesa.edu
Deleting well-known route.
Adding well-known route.
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.
Processing route /oapi/v1/namespaces/default/routes/openshift-cloud-console with domain cloud.crl.coloradomesa.edu.
unable to load certificate
139644322670496:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE
Getting new certificate for cloud.crl.coloradomesa.edu
Deleting well-known route.
Adding well-known route.
2017/07/04 19:42:54 [error] 7#0: *5 directory index of "/opt/app-root/src/html/.well-known/acme-challenge/" is forbidden, client: 10.128.0.1, server: _, request: "GET /.well-known/acme-challenge/ HTTP/1.1", host: "cloud.crl.coloradomesa.edu"
2017/07/04 19:43:03 [error] 7#0: *6 directory index of "/opt/app-root/src/html/.well-known/acme-challenge/" is forbidden, client: 10.128.0.1, server: _, request: "GET /.well-known/acme-challenge/ HTTP/1.1", host: "cloud.crl.coloradomesa.edu"
Route letsencrypt-cloud.crl.coloradomesa.edu not admitted.
Do i need to generate the route with a self-signed cert first or can I leave the fields blank?
So the route works fine. Can you please oc rsh
to the pod and try the curl from there?
Aha! It was another configuration problem, but this time i'll tell everyone what it was:
Internal dns didnt point the base subdomain (in this case cloud.crl.coloradomesa.edu) to the master, only the external dns was. It's working now, and I found out by running curl inside the container like @ibotty said. I didnt think about whether the container was trying to contact itself.
@ibotty thanks for helping me troubleshoot!
Great you got it to work. That seems a frequent problem. I (or some other generous person) ought to write some troubleshoot section in the Readme.
Also i'd like to add: Do not set any routes to redirect insecure traffic. Make the routes allow insecure so that the acme bot can use http. It was stalling on route not admitted until i changed my routes to allow. The redirect insecure applies for all routes on the domain, so it affects the acme-challenge route.
Yes, that ought to be documented. That's a limitation of openshift's router. These routes always come first.
Upon investigating, I cannot reproduce the problem you describe with routes. Can you be more specific (or open another bug report)?
In short, with a edge-terminated route with either Redirect, Allow or no insecure policy, I can successfully use an Allow-edge-terminated path-restricted route.
So it is expected that a redirect HTTP to HTTPS route should work? I'm getting Route letsencrypt-mydomain not admitted.
https://mydomain/.well-known/acme-challenge will give a cert error, if bypassed it gives Application is not available
http://mydomain/.well-known/acme-challenge will redirect to https.
Also it does work without the redirect.
If http://mydomain/.well-known/acme-challenge redirects, openshift's router does not use the create route. Can you please recreate the route (maybe manually) and paste the (redacted) output of oc get route -o yaml
? Most routes I have are edge-terminated routes with redirect insecure policy, so I am positive that setup should work.
This is a fresh deployment, fresh cluster, followed all directions exactly and i get a result similar to #21.
I don't even know what the solution was to #21. Afaik the roles are working correctly. I'm not rate-limited either. It just hasn't been able to request any certificates atall.
Here is log output of the watcher container:
this repeats continuously until i destroy the deployment. I've recreated the deployment numerous times to no avail.
I also created an account-key with dehydrated and saved it as a secret, which is loaded into the pod.