Problems with SSL passthrough when using SNI

[TambetP]

Description

I need to pass SSL to different nginx proxys in docker swarm. Im trying to set it up but having problems.

Steps to reproduce the issue:

1. Deploy DFP
2. Deploy different nginx https proxys with different certificates with sni configuration in docker-compose.
3. Deploy a service behind nginx proxy
4. curl that service

Describe the results you received: Some of the requests go through some get Docker Flow Proxy: 503 Service Unavailable.

Proxy log: 2018/07/04 11:17:02 HAPRoxy: 172.28.22.175:61976 [04/Jul/2018:11:17:01.573] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/469 3806 -- 1/1/0/0/0 0/0 2018/07/04 11:17:03 HAPRoxy: 172.28.22.175:61977 [04/Jul/2018:11:17:03.044] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:04 HAPRoxy: 172.28.22.175:61978 [04/Jul/2018:11:17:04.214] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:05 HAPRoxy: 172.28.22.175:61979 [04/Jul/2018:11:17:05.258] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:06 HAPRoxy: 172.28.22.175:61980 [04/Jul/2018:11:17:06.273] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:07 HAPRoxy: 172.28.22.175:61981 [04/Jul/2018:11:17:07.251] services~ services/ -1/-1/-1/-1/1 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:08 HAPRoxy: 172.28.22.175:61982 [04/Jul/2018:11:17:08.244] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" 2018/07/04 11:17:09 HAPRoxy: 172.28.22.175:61983 [04/Jul/2018:11:17:08.903] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/431 3807 -- 1/1/0/0/0 0/0 2018/07/04 11:17:10 HAPRoxy: 172.28.22.175:61984 [04/Jul/2018:11:17:09.911] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 8/1/426 3806 -- 1/1/0/0/0 0/0 2018/07/04 11:17:11 HAPRoxy: 172.28.22.175:61985 [04/Jul/2018:11:17:10.901] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/0/423 3806 -- 1/1/0/0/0 0/0 2018/07/04 11:17:12 HAPRoxy: 172.28.22.175:61986 [04/Jul/2018:11:17:11.871] service_443 test1_nginx-consul-proxy-be1443_0/test1_nginx-consul-proxy 7/1/429 3806 -- 1/1/0/0/0 0/0 2018/07/04 11:17:13 HAPRoxy: 172.28.22.175:61987 [04/Jul/2018:11:17:13.123] services~ services/ -1/-1/-1/-1/0 503 674 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" (edited)

Requests work 100% when done from inside DPF container.

Additional information you deem important (e.g. issue happens only occasionally): Nginx labels: com.df.notify: 'true' com.df.pathType: "req_ssl_sni -i -m end" com.df.servicePath: ".test1.company.xyz,.test1.companyassets.xyz" com.df.srcPort: 443 com.df.reqMode: sni com.df.port: 1443

Proxy config: frontend service_443 bind *:443 mode tcp option tcplog log global tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 } acl sni_test1_nginx-consul-proxy1443-1 req_ssl_sni -i -m end .test1.company.xyz req_ssl_sni -i -m end .test1.companyassets.xyz acl srcPort_test1_nginx-consul-proxy443_0 dst_port 443 use_backend test1_nginx-consul-proxy-be1443_0 if sni_test1_nginx-consul-proxy1443-1 srcPort_test1_nginx-consul-proxy443_0

backend test1_nginx-consul-proxy-be1443_0 mode tcp server test1_nginx-consul-proxy test1_nginx-consul-proxy:1443

Certificates are mounted to the proxy at /certs Wildcard letsencypt certificates to .test1.company.xyz and .test1.companyassets.xyz domains.

DFP version 18.07.03-68

[TambetP]

Update: I tried out https://github.com/sepich/dockergen-sni Modified it a little and it seems to work. Have not been able to fully test it but the HA proxy config looks like: frontend ft_sni_vip bind *:443 mode tcp tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 }

acl acls_0ae16ad680f7a4d9c176a10e65cf2050df9c17be req_ssl_sni -i -m end .test1.company.xyz req_ssl_sni -i -m end .test1.companyassets.xyz use_backend bk_ssl_test1_ncp if acls_0ae16ad680f7a4d9c176a10e65cf2050df9c17be

backend bk_ssl_test1_ncp mode tcp balance roundrobin stick-table type binary len 32 size 30k expire 30m acl clienthello req_ssl_hello_type 1 acl serverhello rep_ssl_hello_type 2 tcp-request inspect-delay 5s tcp-request content accept if clienthello tcp-response content accept if serverhello stick on payload_lv(43,1) if clienthello stick store-response payload_lv(43,1) if serverhello server test1_nginx-consul-proxy.1.za4jjp2q7nmekcq7mfw4wwmkj_0_0 10.255.1.127:1443 check check-ssl inter 10s server test1_nginx-consul-proxy.1.za4jjp2q7nmekcq7mfw4wwmkj_0_1 10.0.1.20:1443 check check-ssl inter 10s

[thomasjpfan]

1. How are you creating the DFP service?
2. What version of docker are you running on?

[TambetP]

1. DFP is created as a stack. 3 listeners 1 on each manager and 1 proxy. Config is basicly a copy-paste from https://proxy.dockerflow.com/swarm-mode-stack/
2. Docker version: 17.06.2-ee-6

[thomasjpfan]

It looks like dockergen-sni used the IP address of your service. It is possible this is an overlay issue. Can you try setting com.df.discoveryType=DNS on your service?

[TambetP]

Adding com.df.discoveryType=DNS does not seem to change anything. Also when I make requests inside DFP container to the nginx(using container alias in overlay network not ip) everything works perfectly.

[thomasjpfan]

Functionally, there are only two differences between the working dockergen-sni config and the DFP config. DFP uses dst_port 443, and DFP has one backend service. Can you provide the following pieces of info:

1. The complete config of the DFP configuration with the defaults.
2. The stack files or docker commands you are using.

[TambetP]

Yes sure.

DFP docker-compose https://docs.google.com/document/d/1wjVYOdWm2fykz6maXKIzpF__2Qp3EqjvHA6m-qgl1jo/edit?usp=sharing

Nginx docker-compose https://docs.google.com/document/d/1_dk_1Q7-kpctFZyeGY5ZwA9ivWLvre4CbfLD7vflpm0/edit?usp=sharing

Haproxy config https://docs.google.com/document/d/10lQL3IV1J8vmZMVP5FDrrZlmtN-mL7O9Sdojg0NBf_Y/edit?usp=sharing

[thomasjpfan]

There are two bind *:443 in the haproxy config. Try setting the env variable DEFAULT_PORTS=80 in DFP. This will configure DFP to only bind 80 in frontend services.

[TambetP]

Thank you very much! Seems to work. Ill let you know if any problems come up