Add libssl-dev to Dockerfile

benjamin-awd commented 3 months ago

Similar to https://github.com/ArroyoSystems/arroyo/pull/609 this fixes a bug that occurs at runtime when attempting to compile a UDF.

The following error was resolved by installing libssl-dev, based on this thread.

    Checking chrono v0.4.38
   Compiling openssl-sys v0.9.102
   Compiling slab v0.4.9
   Compiling lexical-write-integer v0.8.5
error: failed to run custom build command for `openssl-sys v0.9.102`

Caused by:
  process didn't exit successfully: `/tmp/arroyo/udf_build/target/release/build/openssl-sys-91ac48530782f856/build-script-main` (exit status: 101)
  --- stdout
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  run pkg_config fail: 
  pkg-config exited with status code 1
  > PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 pkg-config --libs --cflags openssl

  The system library `openssl` required by crate `openssl-sys` was not found.
  The file `openssl.pc` needs to be installed and the PKG_CONFIG_PATH environment variable must contain its parent directory.
  The PKG_CONFIG_PATH environment variable is not set.

  HINT: if you have installed the library, try setting PKG_CONFIG_PATH to the directory containing `openssl.pc`.

  --- stderr
  thread 'main' panicked at /usr/local/cargo/registry/src/index.crates.io-6f17d22bba15001f/openssl-sys-0.9.102/build/find_normal.rs:190:5:

  Could not find directory of OpenSSL installation, and this `-sys` crate cannot
  proceed without this knowledge. If OpenSSL is installed and this crate had
  trouble finding it,  you can set the `OPENSSL_DIR` environment variable for the
  compilation process.

  Make sure you also have the development packages of openssl installed.
  For example, `libssl-dev` on Ubuntu or `openssl-devel` on Fedora.

  If you're in a situation where you think the directory *should* be found
  automatically, please open a bug at https://github.com/sfackler/rust-openssl
  and include information about your system as well as this message.

  $HOST = x86_64-unknown-linux-gnu
  $TARGET = x86_64-unknown-linux-gnu
  openssl-sys = 0.9.102

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

mwylde commented 3 months ago

Hey Ben, thanks for the contribution!

I don't think it makes sense to merge this PR, and unlike #609 I wouldn't consider this a general issue.

Some crates have dependencies on system libraries, like libssl in this case. These are shared libraries (typically written in C) that are managed via system package managers rather than cargo. While we can add them to the image, there are hundreds of possible libraries a crate might depend on, so I don't think it makes sense to add all of them for image size and security reasons.

If you need to use a crate that relies on a system library that's not installed by default, you have two options:

In this case, openssl-sys supports vendoring, which means that the crate can build and statically link the c library into the binary (you can enable that with features = ["vendored"]). Many other such crates also support vendoring.
Otherwise, you can build a custom image off of the official one that includes the extra packages you need

In the future, I could imagine a more general solution that allows you to specify system packages you need as well.

benjamin-awd commented 3 months ago

In this case, openssl-sys supports vendoring, which means that the crate can build and statically link the c library into the binary (you can enable that with features = ["vendored"]). Many other such crates also support vendoring.

This is pretty cool, but unfortunately ran into another error 'Can't locate FindBin.pm in @INC' (I think the C library expects perl to be installed)

We're going to take your latter suggestion and maintain a custom image

ArroyoSystems / arroyo

Add libssl-dev to Dockerfile #647