OpenSSL upgrade broke postgres

[Diggsey]

It appears that the recent upgrade of OpenSSL broke our builds which link to pq-sys.

  = note: /home/rust/project/target/x86_64-unknown-linux-musl/release/deps/libpq_sys-ceff9b8e9ad5ca28.rlib(fe-secure-openssl.o): In function `initialize_SSL':
          fe-secure-openssl.c:(.text+0x102): undefined reference to `SSLv23_method'
          /home/rust/project/target/x86_64-unknown-linux-musl/release/deps/libpq_sys-ceff9b8e9ad5ca28.rlib(fe-secure-openssl.o): In function `pgtls_verify_peer_name_matches_certificate_guts':
          fe-secure-openssl.c:(.text+0x131d): undefined reference to `sk_num'
          fe-secure-openssl.c:(.text+0x1342): undefined reference to `sk_value'
          fe-secure-openssl.c:(.text+0x138b): undefined reference to `sk_free'
          fe-secure-openssl.c:(.text+0x13c7): undefined reference to `sk_free'
          /home/rust/project/target/x86_64-unknown-linux-musl/release/deps/libpq_sys-ceff9b8e9ad5ca28.rlib(fe-secure-openssl.o): In function `pgtls_init':
          fe-secure-openssl.c:(.text+0x14b9): undefined reference to `CRYPTO_get_id_callback'
          fe-secure-openssl.c:(.text+0x14c7): undefined reference to `CRYPTO_get_locking_callback'
          fe-secure-openssl.c:(.text+0x14d6): undefined reference to `CRYPTO_set_locking_callback'
          fe-secure-openssl.c:(.text+0x14e8): undefined reference to `SSL_library_init'
          fe-secure-openssl.c:(.text+0x14ed): undefined reference to `SSL_load_error_strings'
          fe-secure-openssl.c:(.text+0x14f9): undefined reference to `CRYPTO_num_locks'
          fe-secure-openssl.c:(.text+0x1524): undefined reference to `CRYPTO_num_locks'
          fe-secure-openssl.c:(.text+0x1586): undefined reference to `CRYPTO_set_id_callback'
          /home/rust/project/target/x86_64-unknown-linux-musl/release/deps/libpq_sys-ceff9b8e9ad5ca28.rlib(fe-secure-openssl.o): In function `pgtls_close':
          fe-secure-openssl.c:(.text+0x1679): undefined reference to `CRYPTO_get_locking_callback'
          fe-secure-openssl.c:(.text+0x1686): undefined reference to `CRYPTO_get_id_callback'
          fe-secure-openssl.c:(.text+0x1695): undefined reference to `CRYPTO_set_id_callback'
          fe-secure-openssl.c:(.text+0x169e): undefined reference to `CRYPTO_set_locking_callback'
          collect2: error: ld returned 1 exit status

These symbols seem to be deprecated functions that were removed in this version of OpenSSL, so my guess is that the version of libpq installed still relies on those old symbols, and that upgrading libpq should fix the issue.

[Diggsey]

On further inspection, it looks like the issue may be more subtle:

OpenSSL 1.1.0 uses the preprocessor to rename SSLv23_method to TLS_method: https://github.com/openssl/openssl/blob/98278b963171ece10a42d18594045b875103115b/include/openssl/ssl.h#L1905

But libpq is still trying to link to SSLv23_method, which implies it was built against the wrong OpenSSL header files. I'm guessing it's seeing the system OpenSSL headers instead of the ones installed for musl.

[emk]

Oh, I'm sorry that you've hit this problem!

I'm going to write up an overlying detailed answer, one which I hope may cover multiple issues with OpenSSL.

OpenSSL 1.0.2: By any chance, if you still using OpenSSL 1.0.2? If so, you should be aware that 1.0.2 has reached its end-of-life, and will no longer be receiving security updates unless you have a paid corporate support contract with the OpenSSL maintainers.

Note: The latest stable version is the 1.1.1 series. This is also our Long Term Support (LTS) version, supported until 11th September 2023. All other versions (including 1.1.0, 1.0.2, 1.0.0 and 0.9.8) are now out of support and should not be used. Users of these older versions are encourage to upgrade to 1.1.1 as soon as possible. Extended support for 1.0.2 to gain access to security fixes for that version is available.

If you do need 1.0.2 to build, there are still specific 1.0.2 images available as described in the README. You may want to try using one of the following:

FROM ekidd/rust-musl-builder:1.42.0-openssl10

FROM ekidd/rust-musl-builder:nightly-2020-03-12-openssl10

These will not be updated for 1.43.0 or newer nightly releases, because 1.0.2 is not longer receiving security updates. You should still be able to build newer releases yourself, however, if you have an OpenSSL commercial support policy, using hooks/build and hooks/test. You will need to set DOCKER_TAG=1.43.0-openssl10 IMAGE_NAME=$YOUR_ORG/rust-musl-builder.

OpenSSL 1.1.1. If you have already migrated to OpenSSL 1.1.1, and recent changes broke your build, then please try checking out the repository and running ./test-image. This will build a fresh image, and use it to compile and link a variety of real apps, including examples/using-diesel, that relies on libpq.

Before every image release, we compile examples/using-diesel automatically, and if it does not compile and link, then we don't that image.

So if examples/using-diesel fails for you, please let me know. If it succeeds, then we can try to figure out how it differs from your application.

[Diggsey]

It turns out this problem was caused by the build cache in our CI, and the fact that the pq-sys crate embeds libpq.a inside its .rlib.

A previous CI run used the prior version of rust-musl-builder to compile our applicaiton (including pq-sys.rlib). The later runs restored this build cache, and because the pq-sys dependency had not changed, it reused the same .rlib. The problem is that this .rlib had the version of pq-sys built against OpenSSL 1.0, whereas the rest of our application was now being built against OpenSSL 1.1.

[emk]

Glad to hear you figured it out! I hated forcing then OpenSSL 1.0.2 -> 1.1.1 upgrade for people using the default image, because knew it was going to break builds. But now that 1.0.2 no longer receives security updates, I don't feel comfortable providing it.

(Also, I've been building the -openssl11 images since April of 2019, and it was probably time to finish the switchover.)