`openvpn exec` with management interface doesn't work with openssl 1.1.1

[dpb587]

When running openvpn exec, connections may fail with the following error:

$ ssoca openvpn exec
Thu Mar  7 22:20:02 2019 OpenVPN 2.4.7 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Feb 20 2019
Thu Mar  7 22:20:02 2019 library versions: OpenSSL 1.1.1a  20 Nov 2018, LZO 2.10
Thu Mar  7 22:20:02 2019 MANAGEMENT: Connected to management server at [AF_INET]127.0.0.1:37153
Thu Mar  7 22:20:02 2019 MANAGEMENT: CMD 'certificate'
Thu Mar  7 22:20:03 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]192.0.2.1:1194
Thu Mar  7 22:20:03 2019 Socket Buffers: R=[87380->87380] S=[16384->16384]
Thu Mar  7 22:20:03 2019 Attempting to establish TCP connection with [AF_INET]192.0.2.1:1194 [nonblock]
Thu Mar  7 22:20:04 2019 TCP connection established with [AF_INET]192.0.2.1:1194
Thu Mar  7 22:20:04 2019 TCP_CLIENT link local: (not bound)
Thu Mar  7 22:20:04 2019 TCP_CLIENT link remote: [AF_INET]192.0.2.1:1194
Thu Mar  7 22:20:04 2019 TLS: Initial packet from [AF_INET]192.0.2.1, sid=9ec63bb4 f2d748f6
Thu Mar  7 22:20:04 2019 VERIFY OK: depth=1, CN=ssoca
Thu Mar  7 22:20:04 2019 VERIFY KU OK
Thu Mar  7 22:20:04 2019 Validating certificate extended key usage
Thu Mar  7 22:20:04 2019 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Thu Mar  7 22:20:04 2019 VERIFY EKU OK
Thu Mar  7 22:20:04 2019 VERIFY OK: depth=0, C=USA, O=Cloud Foundry, CN=openvpn
Thu Mar  7 22:20:04 2019 OpenSSL: error:04066076:rsa routines:rsa_ossl_private_encrypt:unknown padding type
Thu Mar  7 22:20:04 2019 OpenSSL: error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib
Thu Mar  7 22:20:04 2019 TLS_ERROR: BIO read tls_read_plaintext error
Thu Mar  7 22:20:04 2019 TLS Error: TLS object -> incoming plaintext read error
Thu Mar  7 22:20:04 2019 TLS Error: TLS handshake failed
Thu Mar  7 22:20:04 2019 Fatal TLS error (check_tls_errors_co), restarting
Thu Mar  7 22:20:04 2019 SIGHUP[soft,tls-error] received, process restarting

It appears to be related to the management interface providing signing operations because when using static certificates, it works.

This has been observed on Debian Buster and can be reproduced with the debian:buster image. This problem was originally misidentified as a bug with Buster's openvpn/openssl packages.

As a workaround, the --static-certificate option can be used:

$ ssoca openvpn exec --static-certificate

For later laziness...

$ docker run -e SSOCA_ENVIRONMENT=default --privileged -ti debian:buster
apt update
apt install -y openvpn wget
wget -O /usr/local/bin/ssoca https://github.com/dpb587/ssoca/releases/download/v0.13.0/ssoca-client-0.13.0-linux-amd64
echo "ab7b8de0878b8b70664748a01d1f4742c634944e  /usr/local/bin/ssoca" | sha1sum -c
chmod +x /usr/local/bin/ssoca
ssoca env set https://ssoca.example.com
ssoca openvpn exec

[dpb587]

cc @mhoran

[dpb587]

I've spent several debugging sessions looking into this, but haven't made any progress. Until more users or environments run into this to make it a higher priority, I'm not going to continue spending time on it. Anyone else, please feel free!

The closest thing I can find is that, for some reason, openvpn is not sending the RSA_SIGN request to the management interface. In a successful connection, NEED-CERTIFICATE is sent to the management interface before connecting to the remote OpenVPN server; then the RSA_SIGN messages normally occurs in place of the two OpenSSL error messages which appear after the successful VERIFY messages. In this case, NEED-CERTIFICATE occurs, but RSA_SIGN does not.

Seems reasonable that if openvpn is passing an empty RSA signature to libssl, those two errors would show up and cause a failure.

Debugging thoughts:

• haven't tried on other versions of Debian
• compiling openvpn from source has the same issue
• changing order of --management-external-key and --management-external-cert does not help
• if you use ssoca --log-level=DEBUG openvpn exec the management interface now logs messages it receives (recent change)
• if you use ssoca openvpn exec -- --verb 9 you'll see much more debug output from openvpn itself
• for smaller repros or reducing an upstream bug, might want to use nc or a lightweight wrapper to prove/disprove openvpn is not making the correct management calls and rule out anything with ssoca

[dpb587]

This continues to be an issue, recently encountered by a couple other teams.

[dpb587]

This is now happening with the latest Homebrew build of openvpn which more people are likely to notice. Need to revisit this soon.

[dpb587-pivotal]

Spent some time investigating, but no proper explanation or resolution. Created an upstream ticket with openvpn at https://community.openvpn.net/openvpn/ticket/1216. Hopefully someone there will be able to provide more insight.

[dpb587]

Upstream has responded acknowledging the bug:

This is from OpenSSL 1.1.1 defaulting to PSS padding (for TLS 1.2 and 1.3) which it does internally and requests a non-padded signature. But we currently have an implicit assumption that RSA signature from management will use PKCS1_RSA_PADDING. This makes it impossible to prompt for signature when 1.1.1 is in use.

There is a patch to fix this (​https://patchwork.openvpn.net/patch/587/) which is under review and requires some more tweaks.

The patch was started about a year ago, but the conversation seemed to stop. It was restarted from due to the ticket, but, given its history and it requiring a management API change I would guess the change will take a while to land in an official version and then OS packages.

So, in the meantime, I've added an internal, automatic workaround for this by trying to detect the library version that openvpn is using and then falling back to --static-certificates-behavior for v1.1.1. The upcoming version 0.18+ should include this.

I'll keep this issue open for continued tracking of the issue.