Open richm opened 7 years ago
@richm Thanks for your feedback. I have updated the caddy-docker image to make it require clientca, the Caddyfile looks like below:
:8443 {
tls /srv/certs/example_wildcard.crt /srv/certs/example_wildcard.key {
clients require /srv/certs/ca.pem
}
root /srv/publics
browse /test
}
:8080 {
root /srv/public
browse /test
}
And after I create reencrypt route with the image above. The haproxy.config shows as the same what you have pasted.
backend be_secure_u1p1_route-reencrypt
mode http
option redispatch
balance leastconn
timeout check 5000ms
http-request set-header X-Forwarded-Host %[req.hdr(host)]
http-request set-header X-Forwarded-Port %[dst_port]
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header Forwarded for=%[src];host=%[req.hdr(host)];proto=%[req.hdr(X-Forwarded-Proto)]
cookie d2414779495cf6fbcf38bab93d1053a9 insert indirect nocache httponly secure
server 7b6079c65fe6fc12f9d8213b323cceb2 10.129.0.196:8443 ssl check inter 5000ms verify required ca-file /var/lib/haproxy/router/cacerts/u1p1_route-reencrypt.pem cookie 7b6079c65fe6fc12f9d8213b323cceb2 weight 100
And when I try to access the backend via the route, I got failed.
$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem --cert example_wildcard.pem --key example_wildcard.key
curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
When I test the image on local, it works.
$ curl https://www.example.com:8443/ --cacert certs/ca.pem --cert certs/example_wildcard.pem --key certs/example_wildcard.key
Hello-OpenShift-1 https-8443
Looks like the --cert
and --key
parameter is required when the clients require /srv/certs/ca.pem
added to the caddyfile.
I am not sure if this is the bug you mentioned in the router code.
Here is the definition of the reencrypt route. https://docs.openshift.org/latest/architecture/core_concepts/routes.html#secured-routes
If I got your point correctly, I think I can file bug or contact the devel for help.
$ curl --resolve sticky-reen.example.com:443:10.66.140.165 https://sticky-reen.example.com --cacert ca.pem curl: (51) Unable to communicate securely with peer: requested domain name does not match the server's certificate.
What is the sticky-reen.example.com server cert subject DN? subjectAltName?
Looks like the --cert and --key parameter is required when the clients require /srv/certs/ca.pem added to the caddyfile.
I think it was really my misunderstanding of how to make the re-encrypt route work when the server requires client cert. I could not figure out how to configure the route to use client cert auth when re-encrypting the connection from the router to the server. That should be an RFE for OpenShift and/or Kubernetes.
Here is the server cert info:
$ openssl x509 -in example_wildcard.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 14 (0xe)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=SC, L=Default City, O=Default Company Ltd, OU=Test CA, CN=www.exampleca.com/emailAddress=example@example.com
Validity
Not Before: May 25 08:12:58 2016 GMT
Not After : May 23 08:12:58 2026 GMT
Subject: CN=*.example.com, ST=BJ, C=CN/emailAddress=bmeng@redhat.com, O=RH, OU=OS
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (1024 bit)
Modulus:
00:a0:11:cd:c3:b6:d2:57:20:fa:a0:df:7b:dc:40:
11:d7:b6:75:b5:22:8f:d2:d6:72:55:ff:55:f1:a9:
d1:78:72:77:37:ff:38:39:8b:e5:cf:c8:cb:f7:31:
80:ab:e8:a3:b0:d9:4e:e3:d8:b1:7a:d2:ef:3f:6e:
b8:b2:38:4b:8b:2d:aa:3f:e7:db:52:ab:60:8c:03:
74:85:ef:68:e7:3a:09:34:d0:c6:a5:04:db:42:d9:
d3:c2:e4:47:c0:4b:ad:ae:0b:6e:7e:e5:86:54:60:
6b:23:8c:0a:57:dd:50:be:3b:e0:2b:f4:4b:83:a5:
0d:2d:3b:89:d6:c3:d6:45:f3
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Signature Algorithm: sha1WithRSAEncryption
c2:11:81:ef:1d:5d:6c:99:26:cb:de:d5:ac:9c:9e:20:71:77:
25:94:32:83:3d:b4:f0:43:bd:f6:40:94:b2:c3:5a:e6:ee:a1:
a1:17:2e:88:f2:df:01:87:1b:30:c6:39:6a:ef:60:7c:26:3d:
f5:83:35:ff:88:96:bc:5e:b6:22:0e:89:7c:4a:7c:6b:53:b0:
eb:aa:74:90:9c:6f:3d:f2:ef:45:a2:20:2c:e2:5f:99:bb:76:
a7:a8:19:f9:c6:85:fe:98:b3:af:cb:d3:f8:5e:70:72:f1:c3:
ce:91:96:19:49:9c:7e:d6:d1:e0:77:a9:e4:fb:bd:ce:f2:b4:
3c:5c:44:59:6f:84:5c:60:00:3d:ce:17:51:fc:5c:a2:56:b9:
a8:21:ce:eb:34:ac:33:e5:42:f8:84:3d:89:18:7e:65:01:30:
d2:e5:fd:d6:d6:11:53:55:df:00:9a:91:a6:8c:a1:88:35:db:
2e:29:d5:17:78:a6:05:b7:0f:05:d6:61:b9:c0:5f:92:b5:4d:
bb:6e:00:b5:af:9a:23:88:70:66:ed:2d:3c:0a:a8:8e:a9:74:
f2:71:e1:ba:3e:cd:bc:0e:ba:8f:d0:f6:56:e3:45:c6:46:00:
4a:db:0c:a0:38:39:fa:53:0a:ba:6e:fe:a0:03:5d:f9:c9:54:
60:ae:59:4a
I think there is a problem with the cert.
CN=*.example.com, ST=BJ, C=CN/emailAddress=bmeng@redhat.com, O=RH, OU=OS
It looks as though this is in the wrong order. For one, the order is different than the Issuer, which has CN as the righmost component. For another, here is what a working server cert looks like:
C=US, ST=NM, L=ABQ, O=CO, CN=f13x8664.testdomain.com
It is the CN value that the client side uses to check the server cert for MITM. This CN value should be the rightmost value in the Subject in order for the client to be able to use it for checking.
@richm how do you obtain info about your cert? It seems that openssl text output does not necessarily match actual order. It can change using different options. For example:
$ openssl x509 -in example_wildcard.pem -text -noout -nameopt RFC2253
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 14 (0xe)
Signature Algorithm: sha1WithRSAEncryption
Issuer: emailAddress=example@example.com,CN=www.exampleca.com,OU=Test CA,O=Default Company Ltd,L=Default City,ST=SC,C=US
Validity
Not Before: May 25 08:12:58 2016 GMT
Not After : May 23 08:12:58 2026 GMT
Subject: OU=OS,O=RH,emailAddress=bmeng@redhat.com,C=CN,ST=BJ,CN=*.example.com
<...>
I Believe @jianlinliu has generated the certificates by OpenSSL and I doubt format is invalid. What I would suspect is that the route is not using the desired certificate for some reason in the example above.
One can see actual certificate returned by server in two easy ways:
$ curl -v https://google.com
<...>
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.google.com,O=Google Inc,L=Mountain View,ST=California,C=US
* start date: Mar 01 14:47:44 2017 GMT
<...>
btw you can see the google cert also appears with CN in the beginning but I believe this is only output formatting. One has to look lower level at cert to check actual order.
Also full server chain can be extracted via openssl
:
# retrieve cert chain
openssl s_client -showcerts -connect www.example.com:443 -prexit </dev/null
# for SNI:
openssl s_client -showcerts -servername www.example.com -connect www.example.com:443 -prexit </dev/null
I think first thing is to check whether the route is returning the expected certificate to client. And only then check whether client cert validation is performed.
I used openssl x509 -in servercert.pem -text
- in my case, both Issuer and Subject are printed with the CN being the rightmost part of the DN. So I guess that isn't the issue here.
For the router reencrypt test, client certs/keys are allowed, but do not appear to be used. https://github.com/openshift-qe/v3-testfiles/blob/master/routing/reencrypt/route_reencrypt-path.json#L14
This sets up the route with cert and key. However, the /etc/Caddyfile in the caddy-docker pod looks like this:
The Caddy documentation shows this:
I believe
clients
should be configured like this to require client certs:or wherever the example_ca cert is stored.
I think the lack of testing is masking a bug in the router code in openshift, which may have been fixed in 1.4. In 1.3 /var/lib/haproxy/conf/haproxy.config looks like this:
Note the
server
line. I'm not sure which haproxy documentation is applicable to the haproxy version in OSE 3.3. But I did find this: https://www.haproxy.com/doc/aloha/7.0/haproxy/tls.html#configuring-haproxy-for-ssl-tlsI do not see a "crt" directive in the "server" line above. So I don't know how openshift is supposed to use the cert and key specified in the reencrypt route configuration.