travisghansen
commented
3 years ago Welcome! I’m afk at the moment but I’ll have a look early next week and see what can be done.
Closed sddan closed 3 years ago
travisghansen
commented
3 years ago Welcome! I’m afk at the moment but I’ll have a look early next week and see what can be done.
travisghansen
commented
3 years ago Ok so looking this over and there are some issues that leave me reluctant to address. Primarily ingress supports paths (ie: same hostname splitting different paths to different backing services) which I believe will be incompatible with sni/l4 (if I misunderstand something let me know).
Do you use many different domains? Is it possible to ssl terminate (l7) with haproxy and acme directly on pfsense and then route appropriately from there (keeping ssl entirely out of your cluster and non-cluster services)?
duckblaster
commented
3 years ago In the pfsense haproxy config you want to select "ssl/https (tcp mode)" for the front-end type.
That will let you set acl conditions based on sni to select the k8s ingress backend.
If you want pfsense to handle ssl for some sites, make q backend that loops back to a dummy front-end and do the ssl termination there.
I found some docs here: https://github.com/PiBa-NL/pfsense-haproxy-package-doc/wiki/pfsense_2_3_haproxy_sni_plus_offloading_backends
sddan
commented
3 years ago @travisghansen That is certainly a possibility for a lot of the services that are running. However, my preference is to have SSL terminated as close as possible to the service, which reduces the distance packets travel unencrypted. Which is why I prefer the SNI method. I'm not very familiar with paths and how they interact with SNI, but I think you're probably right about not having ingress paths available with this approach. At the moment, I only have 2 different domains, but I'm not 100% certain that this won't change.
@duckblaster Thanks for the info. I set up the primary frontend for SSL passthrough, and I was able to test if this idea would work by running the controller for a short time to create the shared frontends, then scaling down the deployment to zero pods. Then I went into each of the shared frontends and manually changed the ACL rule to HostSNI. Worked as expected.
The setup I had was one frontend with SSL passthrough (the frontend which receives external traffic, and the frontend with cluster ingresses are bound to). The default backend of this SSL passthrough frontend is another frontend in HTTPS offloading mode, which terminates SSL for services which cannot handle SSL on their own.
If I may suggest, the HostSNI ACL type for shared frontends could be a non-default option selected by an annotation, with the path limitations you have mentioned stated in the readme.
travisghansen
commented
3 years ago Can you send over a snippet of the resulting config.xml acl rules (cleanse anything private obviously)? After making the sni rules were the prefix/path rules deleted or were they still in the acl?
sddan
commented
3 years ago I actually don't make use of different paths at the moment, all my services are on different subdomains. I can set something up to test that if you like. Also, what I did to switch the ACLs over to hostSNI was select hostSNI from the dropdown in the frontend, and remove the custom ACL syntax from the value field, leaving only the hostname. The prefix rules were preserved in the acl definition when selecting HostSNI from the dropdown.
The snippet below was pulled from a pfSense config export.
<haproxy>
<ha_backends>
<item>
<name>k3s-http</name>
<desc>HTTP Frontend</desc>
<status>active</status>
<primary_frontend>k3s-https</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6127b3119a13f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls></ha_acls>
<ha_certificates></ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>172.18.0.202_ipv4</extaddr>
<extaddr_port>80</extaddr_port>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems></a_actionitems>
<a_errorfiles></a_errorfiles>
<backend_serverpool>dummy-pf-http-redirect</backend_serverpool>
</item>
<item>
<name>pf-http</name>
<desc>HTTP Frontend</desc>
<status>active</status>
<primary_frontend>k3s-http</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6127b3119a13f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls></ha_acls>
<ha_certificates></ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>wan_ipv4</extaddr>
<extaddr_port>80</extaddr_port>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems></a_actionitems>
<a_errorfiles></a_errorfiles>
<backend_serverpool>dummy-k3s-https-redirect</backend_serverpool>
</item>
<item>
<name>k3s-https</name>
<desc>HTTPS Frontend</desc>
<status>active</status>
<type>https</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6127b3119a13f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls></ha_acls>
<ha_certificates></ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>172.18.0.202_ipv4</extaddr>
<extaddr_port>443</extaddr_port>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems></a_actionitems>
<a_errorfiles></a_errorfiles>
<primary_frontend>k3s-http</primary_frontend>
<backend_serverpool>dummy-pf-https-redirect</backend_serverpool>
</item>
<item>
<name>k3s-prod-default-whoami</name>
<desc>created by kpc - do not edit</desc>
<status>active</status>
<secondary>yes</secondary>
<primary_frontend>k3s-https</primary_frontend>
<ha_acls>
<item>
<name>k3s-prod-default-whoami-rule-0</name>
<expression>ssl_sni_contains</expression>
<value>whoami.okl</value>
<backendservercountbackend>acme-http</backendservercountbackend>
<_index></_index>
</item>
</ha_acls>
<a_actionitems>
<item>
<action>use_backend</action>
<acl>k3s-prod-default-whoami-rule-0</acl>
<use_backendbackend>k3s-prod-ingress-https</use_backendbackend>
<_index></_index>
</item>
</a_actionitems>
<ha_certificates></ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr></a_extaddr>
<a_errorfiles></a_errorfiles>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6129fa44979f1</ssloffloadcert>
<advanced></advanced>
</item>
<item>
<name>pf-https</name>
<desc>pfSense HAProxy HTTPS</desc>
<status>active</status>
<primary_frontend>k3s-http</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6127b3119a13f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls></ha_acls>
<ha_certificates>
<item>
<ssl_certificate>6129e8714ed4f</ssl_certificate>
<_index></_index>
</item>
<item>
<ssl_certificate>6129f87a36d6f</ssl_certificate>
<_index></_index>
</item>
<item>
<ssl_certificate>6129fa44979f1</ssl_certificate>
<_index></_index>
</item>
<item>
<ssl_certificate>6129f9ebb765f</ssl_certificate>
<_index></_index>
</item>
<item>
<ssl_certificate>6129fa2dd0e72</ssl_certificate>
<_index></_index>
</item>
<item>
<ssl_certificate>612a0a674fa08</ssl_certificate>
<_index></_index>
</item>
</ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>wan_ipv4</extaddr>
<extaddr_port>443</extaddr_port>
<extaddr_ssl>yes</extaddr_ssl>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems></a_actionitems>
<a_errorfiles></a_errorfiles>
</item>
<item>
<name>office</name>
<desc>Office</desc>
<status>active</status>
<secondary>yes</secondary>
<primary_frontend>pf-https</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6129e8714ed4f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls>
<item>
<name>hostmatch-office</name>
<expression>host_matches</expression>
<value>office.okl.ga</value>
<backendservercountbackend>acme-http</backendservercountbackend>
<_index></_index>
</item>
</ha_acls>
<ha_certificates></ha_certificates>
<clientcert_ca></clientcert_ca>
<clientcert_crl></clientcert_crl>
<a_extaddr>
<item>
<extaddr>wan_ipv4</extaddr>
<extaddr_port>80</extaddr_port>
<_index></_index>
</item>
</a_extaddr>
<a_actionitems>
<item>
<action>use_backend</action>
<acl>hostmatch-office</acl>
<use_backendbackend>office</use_backendbackend>
<_index></_index>
</item>
</a_actionitems>
<a_errorfiles></a_errorfiles>
</item>
<item>
<name>this-firewall</name>
<desc>This Firewall - Office</desc>
<status>active</status>
<secondary>yes</secondary>
<primary_frontend>k3s-https</primary_frontend>
<type>http</type>
<httpclose>http-keep-alive</httpclose>
<ssloffloadcert>6129e8714ed4f</ssloffloadcert>
<ssloffloadacl_an>yes</ssloffloadacl_an>
<advanced></advanced>
<ha_acls>
<item>
<name>hostmatch-pfingress</name>
<expression>host_matches</expression>
<value>pfingress.internal.okl.ga</value>
<backendservercountbackend>acme-http</backendservercountbackend>
<_index></_index>
</item>
<item>
<name>hostmatch-pfingress</name>
<expression>ssl_sni_contains</expression>
<value>pfingress.internal.okl.ga</value>
<backendservercountbackend>acme-http</backendservercountbackend>
<_index></_index>
</item>
</ha_acls>
<ha_certificates>
</ha_certificates>
<clientcert_ca>
</clientcert_ca>
<clientcert_crl>
</clientcert_crl>Yeah my concern is more about following/supporting the spec properly in case others use it with that expectation.
I’ll have a look though and see what I can discover.
travisghansen
commented
3 years ago OK team. I've updated to support this scenario (I think). Give it try and let me know if the expected outcome works as desired.
v0.5.6 has added support
A couple notes:
sddan
commented
3 years ago I've tested the new build with my cluster, and it works as expected. Maybe others can chime in with other cluster configurations. For what its worth, I did not set up a new cluster from scratch to test. I just pulled down the new container image to my existing cluster.
From what I understand, SSL offloading only needs to be selected on a http/https frontend, which indicates that pfsense should handle the SSL termination. So it should be unchecked for SSL passthrough. In my own config, I also do not select the SSL offloading checkbox on the frontend that kpc binds frontends to.
travisghansen
commented
3 years ago Ok if you left it unchecked and it worked then it should work either way. Thanks! I’ll document etc now.
travisghansen
commented
3 years ago I've locked in the rules to more strictly follow the spec for both paths and hosts. Can you try v0.5.7 and let me know if the sni rules work?
EDIT: use v0.5.8
sddan
commented
3 years ago Tried it with v0.5.8, the SNI rules work as expected. Thanks for your support on this!
travisghansen
commented
3 years ago Awesome! I'll close this out now.
Hi there,
I'm running cert-manager in my Kubernetes cluster, which manages TLS certs for cluster services. Non cluster services are also running, with pfSense managing the certs for those. Therefore, I would like to be able to have pfSense HAProxy use SSL Passthrough mode for the frontend which cluster services attach to, and select the K8s cluster ingress controller backend via HostSNI, instead of HTTP Host Header, which seems to be the default selector. Is there a way this could be done?