Trouble accessing a service using a Let's Encrypt wildcard certificate

[till]

Description

Trying to access an external server from a flatcar node fails when the external host is using a let's encrypt wildcard certificate.

Impact

We are working on a new service which needs to request data from another which is using a Let's Encrypt wildcard certificate. This is the first time we are using one of those wildcard certs (running into ratelimits otherwise). Since the request does not work, the application is broken.

Environment and steps to reproduce

1. Set-up: Flatcar Linux on OpenStack
2. Task: curl host, also tried to do it with other means — yields same result.
3. Action(s): a. logged into server b. attempted to run curl command
4. Error:

openssl s_client -showcerts -connect sub.example.dev:443
CONNECTED(00000003)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 327 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

No chain of certificates found?

curl output:

core@node-004 ~ $ curl -v https://sub.example.dev
*   Trying a.b.c.d:443...
* Connected to sub.example (a.b.c.d) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: Connection reset by peer in connection to sub.example.dev:443 
* Closing connection 0
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to sub.example.dev:443 

Expected behavior

Request succeeds.

Additional information

NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3374.2.3
VERSION_ID=3374.2.3
BUILD_ID=2023-01-20-1751
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3374.2.3 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3374.2.3:*:*:*:*:*:*:*"

OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)

• I ran sudo update-ca-certificates.
• I was also able to verify that this only happens on Let's Encrypt wildcard certificates. Other certificates are not impacted.
• I also ran an SSLlabs test on the domain which produces an A+ rating.
• All commands shared work on OSX. Also ran them on a CentOS7 server and they work as well. A bit baffled.

[till]

In addition, this is how we request the wildcard certificate with cert-manager:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  namespace: cert-manager
  name: wildcard-example-dev-certificate
spec:
  secretName: wildcard-example-dev
  issuerRef: 
    kind: Issuer
    name: letsencrypt-autodns
  commonName: "example.dev"
  dnsNames:
    - "example.dev"
    - "*.example.dev"

[tormath1]

Hi @till,

Thanks for the bug report. Usually err 104 means that the connection has been reset by peer (as stated in the curl output). Did you try to assert that you can effectively connect to 443 on sub.example.dev with telnet for example from the toolbox or any Docker container?

[till]

Sorry, trying to not incude IPs. Let me repost comment.

[till]

@tormath1 I think I can, but I am not sure:

nc -vv sub.example.dev 443
Ncat: Version 7.92 ( https://nmap.org/ncat )
NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:80000002:system library::No such file or directory
libnsock nsock_iod_new2(): nsock_iod_new (IOD #1)
libnsock nsock_connect_tcp(): TCP connection requested to A.B.C.D:443 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [A.B.C.D:443]
Ncat: Connected to A.B.C.D:443.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #2)
libnsock nsock_read(): Read request from IOD #1 [A.B.C.D:443] (timeout: -1ms) EID 18
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26

I guess for context: The host/IP is exposed via a floating IP on the same k8s cluster. But I need to connect using that instead of some internal service. But maybe it's using the wrong bundle? I am a bit puzzled.

[till]

It is a .dev domain, so SSL is required.

[jepio]

When I spin up a Flatcar VM I can correctly connect to your server with openssl s_client -connect A.B.C.D:443 (caught the IP before the edit). This feels like a networking issue in your setup.

[tormath1]

When I spin up a Flatcar VM I can correctly connect to your server with openssl s_client -connect A.B.C.D:443 (caught the IP before the edit). This feels like a networking issue in your setup.

That's what I wanted to verify before investigating further as I can access another resource using LE wildcard certificates from a Flatcar instance without any issue.

[till]

Any idea how I can troubleshoot this further? We are using the standard cloud controller setup (with CSI, and Octavia, etc.).

[jepio]

Unfortunately I don't know. It could be a firewalling issue but there are too many parameters/ways this could be going wrong. The netcat output is a give away - the connection is being closed (read returns 0 bytes) before any TLS negotiation happens.

[till]

@jepio yeah, you are correct. I just tested this with another app (not using the wildcard). So essentially, I have a bunch of pods running on it, they all get exposed by a loadbalancer which is used by the haproxy ingress and the connection is dropped as soon as I request them via that loadbalancer. Sorry for the noise. :(