Closed hedefalk closed 8 years ago
The first log line is from your plugin I see and seems to indicate "happy path", right? But the second one says it still serves a 502. I guess there's something funky with my haproxy.cfg anyways. I'm pretty new to it… I might as well post the entire thing here if you can spot something:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
# https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
tune.ssl.default-dh-param 2048
lua-load /etc/haproxy/acme-http01-webroot.lua
defaults
log global
mode http
option forwardfor
option http-server-close
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend http
mode http
bind *:80
# Letsencrypt: https://github.com/janeczku/haproxy-acme-validation-plugin
acl url_acme_http01 path_beg /.well-known/acme-challenge/
http-request use-service lua.acme-http01 if METH_GET url_acme_http01
#redirect scheme https code 301 if { hdr(Host) -i repo.woodenstake.se } !{ ssl_fc }
#redirect scheme https code 301 if { hdr(Host) -i jenkins.woodenstake.se } !{ ssl_fc }
frontend https
mode http
bind *:443 ssl crt /etc/haproxy/startssl2.pem
#bind *:443 ssl crt /etc/letsencrypt/live/jenkins.woodenstake.se/haproxy.pem crt /etc/haproxy/startssl2.pem
# Define hosts based on domain names
acl host_jenkins hdr(host) -i jenkins.woodenstake.se
acl host_repo hdr(host) -i repo.woodenstake.se
acl host_jenkins_nas hdr(host) -i jenkins-nas.woodenstake.se
acl host_transmission hdr(host) -i transmission.woodenstake.se
use_backend jenkins if host_jenkins
use_backend nexus if host_repo
use_backend jenkins_nas if host_jenkins_nas
use_backend transmission_nas if host_transmission
backend jenkins
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server jenkins_backend 127.0.0.1:8080
backend nexus
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server nexus_backend 127.0.0.1:8081
backend jenkins_nas
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server jenkins_nas_backend 10.0.1.30
backend transmission_nas
mode http
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server jenkins_nas_backend 10.0.1.30:8181
Feels like a spaghetti of http status codes I see here:
Ok, so I see in the logs that this lua service clearly sets 200. Then I also see a 502 (bad gateway) on the next line in haproxy log. But then the LE client tells me it gets a 400 (bad request). Just doesn't make any sense :´(
If I just put a file there manually and curl it myself I get the same:
~ $ curl -v jenkins.woodenstake.se/.well-known/acme-challenge/eBY2kFvH_GVcBMCMZM10eqEa3wR40M0ZLwy2y6Uk4V8
* Trying 89.253.88.3...
* Connected to jenkins.woodenstake.se (89.253.88.3) port 80 (#0)
> GET /.well-known/acme-challenge/eBY2kFvH_GVcBMCMZM10eqEa3wR40M0ZLwy2y6Uk4V8 HTTP/1.1
> User-Agent: curl/7.41.0
> Host: jenkins.woodenstake.se
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 400 Bad request
< Cache-Control: no-cache
< Connection: close
< Content-Type: text/html
<
<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>
HTTP/1.0 502 Bad Gateway
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
* Closing connection 0
haproxy.log looks pretty much identical…
It don't think it can access the necessary file because of chroot, can you try without it?
@lukastribus Pretty sure that's not it. I added a log just before writing the body:
applet:send(response)
core.Info("[acme] sent response: " .. response)
and I can see the file content in the haproxy log. So there's something later that makes the request fail.
@hedefalk this looks like something happening after the Lua handler returns. Are you running the latest stable (v1.6.2) ? To debug this, i would recommend starting with a minimal config like this one: https://github.com/janeczku/haproxy-acme-validation-plugin/blob/master/haproxy.cfg.example
@janeczku
Yes, running latest stable 1.6.2.
I found something. I did what you suggested and used your stripped down config. And it worked. I then re-inserted my personal config and tested each change. What makes it break is that I have:
option forwardfor
under defaults. See http://cbonte.github.io/haproxy-dconv/configuration-1.6.html#4-option%20forwardfor.
This since I need "X-Forwarded-For" for most of my backends to tell the "real client IP". I guess what happens is that this directive is applied at the end which is after you already did AppletHTTP:start_response()
and wrote to the body with applet:send(response)
which makes stuff blow silently since you can't set new headers after this:
"AppletHTTP.start_response() This function indicates to the HTTP engine that it can process and send the response headers. After this called we cannot add headers to the response; We cannot use the AppletHTTP:send() function if the AppletHTTP:start_response() is not called."
That's at least a guess.
I guess I could instead move my usage of
option forwardfor
to be under each individual backend and be done. I'll report back!
By the way, yesterday I continued googling for options and found:
https://coolaj86.com/articles/lets-encrypt-with-haproxy/
I guess you've seen that? Could I ask if you could elaborate on pros/cons of each approach? He's using the standalone server instead with alternative port. There are options for these under "testing":
--tls-sni-01-port TLS_SNI_01_PORT
Port number to perform tls-sni-01 challenge. Boulder
in testing mode defaults to 5001. (default: 443)
--http-01-port HTTP01_PORT
Port used in the SimpleHttp challenge. (default: 80)
I think the whole thing is pretty confusing - what port to run the standalone server on and what port letsencrypt will challenge you on are obviously two entirely different things. It seems that the way he does it is to set a high port with --tls-sni-01-port
but then he gets the challenge on 443 anyways. I can't seem to read that from the documentation so I'm wandering if this is likely to break in the future.
I moved the option forwardfor
to each individual backend and now I have green beautiful adressbars on all my subdomains :)
Multiply domains on a single run didn't work with the client though. Multiple -d
that is. Don't know if that is a known…
Now I'm gonna look into your script and setup a cronjob. Thanks a lot for this!
@hedefalk
By the way, yesterday I continued googling for options and found: https://coolaj86.com/articles/lets-encrypt-with-haproxy/ I think the whole thing is pretty confusing
Exactly :grin: Thats way i wrote the plugin.
I didn't test multiple domains. Would be interesting to know what exactly doesn't work. Do you still have the client log?
@janeczku Reported here https://github.com/letsencrypt/letsencrypt/issues/1946#issuecomment-165883867
I might have been mistaken - sounds like I might have gotten SAN certificate for all subdomains, but under a folder in webroot for the first subdomain. Maybe. Can't test right now since I've reached some quota.
@hedefalk Use --staging
flag in the letsencrypt
command for testing purposes. There is no rate limit there.
@janeczku But what does that entail? Any downsides? Those are pretty important sites for me… I'm testing but it's still production so to say :)
Use the staging server to obtain test (invalid) certs
Ah then you should not do that! :smile:
Awesome plugin! I've been googling crazy on how to setup letsencypt with my haproxy setup serving multiple sites and this seems to be just what I need.
I'm having problems though:
I have made the changes to haproxy.cfg as per your instructions.
I run the following:
And while it's running, I can ll to see that there is a file appearing briefly where it should:
However, the LE client logs a failure:
If I look at haproxy.log I see:
502 seems to mean can't serve the file? (the file names differ here just because of different test runs, but they are of course the same)
Any idea on what could be wrong? Permissions on the file to serve maybe? Edit: nah, seems to be readable all the way up.
Thanks!