libreSSL support

[cschlack]

crystal version >= 0.22 fails using libressl:

_main.o: In function `add_options':
/usr/lib/crystal/src/openssl/ssl/context.cr:274: undefined reference to `SSL_CTX_set_options'
_main.o: In function `initialize':
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `SSL_get0_param'
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `X509_VERIFY_PARAM_set1_ip_asc'
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `X509_VERIFY_PARAM_set1_host'
collect2: error: ld returned 1 exit status

The problem seem to be here: https://github.com/crystal-lang/crystal/blob/da48f8224825c6a09da5cedba17e673383e1257f/src/openssl/lib_ssl.cr#L5

pkg-config --atleast-version=1.1.0 libssl

This is a check for libssl version >= 1.1.0 libreSSL reports (in my case) version 2.5.4

The libreSSL API follows (asfaik) openssl version 1.0.1

[ysbaddaden]

We use the OpenSSL version numbers to determine which OpenSSL API we should follow. If LibreSSL uses a custom scheme, well... that's complicated :(

[cschlack]

libgit2 (C) uses the following for example:

#if OPENSSL_VERSION_NUMBER < 0x10100000L || defined(LIBRESSL_VERSION_NUMBER)
// this is NOT openssl version 1.1.0
#endif

But there's no way we can access these constants from crystal, right?

Could get the library version using pkg-config --modversion libssl and the make sure the version is between 1.1.0 and 2.0.0 for openssl version 1.1.0 API support?

[ysbaddaden]

Yes, Crystal can't access C headers at compile time. Is libressl installing another .pc file that could be checked for?

[ysbaddaden]

Or what does lib/pkgconfig/openssl.pc says?

[cschlack]

The package installs

/usr/lib/pkgconfig/libtls.pc
/usr/lib/pkgconfig/libcrypto.pc
/usr/lib/pkgconfig/libssl.pc
/usr/lib/pkgconfig/openssl.pc

Contents of /usr/lib/pkgconfig/openssl.pc:

#openssl pkg-config source file

prefix=/usr
exec_prefix=${prefix}
libdir=${exec_prefix}/lib
includedir=${prefix}/include

Name: LibreSSL
Description: Secure Sockets Layer and cryptography libraries and tools
Version: 2.5.4
Requires: libssl libcrypto

[ysbaddaden]

Sigh, that won't help.

[sdogruyol]

Shall we close this or add a label?

[RX14]

perhaps we could run something through the C preprocessor quickly to determine the status of LIBRESSL_VERSION_NUMBER. Or perhaps we should bat this over to the libressl guys, or the distro packagers, to come up with a solution.

[ysbaddaden]

@RX14 the C preprocessor may be the only viable solution.

[ysbaddaden]

Apparently we can use SHLIB_VERSION_NUMBER to detect the API version, so this could help:

OPENSSL_VERSION = {{ `echo "#include <openssl/opensslv.h>\nSHLIB_VERSION_NUMBER" | cc -E -`.chomp.split('\n').last.id }}
puts OPENSSL_VERSION #=> 1.0.0 or 1.1

That doesn't help to detect features introduced in 1.0.2 thought (SHLIB reports 1.0.0), and we can't rely on OPENSSL_VERSION_NUMBER with LibreSSL. We could detect that LIBRESSL_VERSION_NUMBER is defined, then check OPENSSL_VERSION_NUMBER when it's not defined, but it's starting to be very complicated...

[valpackett]

So why not assume that anything above 2.0.0 in pkg-config is LibreSSL?

[LVMBDV]

(Originally suggested by @Vaelatern on voidlinux/void-packages#9831)

LibreSSL has a new TLS library called libtls, different from the OpenSSL-compatible libssl. It could be used to detect if LibreSSL is present on the system. LibreSSL currently doesn't support OpenSSL's libssl API beyond version 1.0.1 so both version checks could be modified to simply check if libtls exists. And that --exists check would be replaced by a --atleast-version check when a new LibreSSL supporting newer OpenSSL APIs is released.

[valpackett]

LIBRESSL_VERSION_NUMBER is better than searching for libtls

[LVMBDV]

Although LibreSSL/OpenSSL version conflict requires a hack to resolve, calling cc to get a version number is too hacky, in my opinion.

[Vaelatern]

cc to get a version number also breaks some cross building setups. A lot of code can be cross compiled, but then some configure script protests because of a trivial test.

[RX14]

@LVMBDV just because libtls is present doesn't mean libssl refers to libressl. You could have a libtls from libressl and a libssl from openssl on the same system.

Yes, calling cc is a hack. If we don't want a hack the libressl guys should come up with a better way of detecting their ABI.

[RX14]

Seems rust-openssl does a similar thing to attempt to detect libressl: parse the headers (we just get gcc to do that for us)

[ysbaddaden]

Calling cc to parse actual C headers is no more a hack than guessing based on the pkg-config version. It's just the same level of issue for cross compilation: we parse a .h file instead of a .pc file (both expected by developers).

Guessing based on the presence of a library file (that we don't even use) is... well... just pure guess.

IMHO: using a C preprocessor is a good solution we have at hand. Just as hacky as guessing from pkg-config.

[vtambourine]

I'm having same problem on Crystal 0.24.1 (2017-12-22). Build an application fails with following error:

O-penS-S-L-5858S-S-L-5858C-ontext5858C-lient.o: In function `add_options':
/usr/lib/crystal/src/openssl/ssl/context.cr:274: undefined reference to `SSL_CTX_set_options'
O-penS-S-L-5858S-S-L-5858S-ocket5858C-lient.o: In function `initialize':
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `SSL_get0_param'
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `X509_VERIFY_PARAM_set1_ip_asc'
/usr/lib/crystal/src/openssl/ssl/socket.cr:15: undefined reference to `X509_VERIFY_PARAM_set1_host'
collect2: error: ld returned 1 exit status

openssl v1.0.2

[aisrael]

Am having the exact same issue as @vtambourine trying to build Amber using ysbaddaden/crystal-alpine as Docker base image.

[ysbaddaden]

May someone that reproduces the issue prepare a pull request to replace pkg-config to use the cc preprocessor command instead, as proposed above? That would be awesome. Thanks!

[LVMBDV]

On it :)

[asterite]

Did anyone even try this on a Mac? I can't compile anything right now:

<stdin>:1:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^~~~~~~~~~~~~~~~~~~~
1 error generated.
Error in line 1: while requiring "./spec/compiler/codegen/macro_spec.cr"

in spec/compiler/codegen/macro_spec.cr:1: while requiring "../../spec_helper"

require "../../spec_helper"
^

in spec/spec_helper.cr:6: while requiring "../src/compiler/crystal/**"

require "../src/compiler/crystal/**"
^

in src/compiler/crystal/tools/playground/agent.cr:1: while requiring "http"

require "http"
^

in src/http.cr:2: while requiring "./http/**"

require "./http/**"
^

in src/http/client.cr:749: expanding macro

{% if !flag?(:without_openssl) %}
^

in macro 'macro_4626924080' /Users/asterite/Projects/crystal/src/http/client.cr:749, line 2:

   1. 
>  2.   require "openssl"

while requiring "openssl"
in src/openssl.cr:1: while requiring "./openssl/lib_ssl"

require "./openssl/lib_ssl"
^

in src/openssl/lib_ssl.cr:1: while requiring "./lib_crypto"

require "./lib_crypto"
^

in src/openssl/lib_crypto.cr:1: expanding macro

{% begin %}
^

in src/openssl/lib_crypto.cr:7: error executing command: echo "#include <openssl/opensslv.h>
OPENSSL_VERSION_NUMBER" | cc  -E -, got exit status 1:

# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 341 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

OPENSSL_VERSION_NUMBER

    OPENSSL_VERSION = {{ system("echo \"#include <openssl/opensslv.h>\nOPENSSL_VERSION_NUMBER\" | " +
                         ^~~~~~

Mac doesn't use OpenSSL, so those headers will never exit there.

How can we fix this?

[ysbaddaden]

Sigh. I'm bored with C bindings. I wish we could just do require "openssl.h" and access whatever on LibC —adding a pass to have libclang parse all headers to resolve all LibC nodes in the AST.

Maybe the current hack can be fixed with a || echo "" so it doesn't fail anymore?

[LVMBDV]

Maybe the current hack can be fixed with a || echo "" so it doesn't fail anymore?

Looks like there was a fallback to true already but I munged it while turning a backtick string to a system() call.

EDIT: Nevermind that || true was for the pkg-config command. You are absolutely right. I am on it.

Mac doesn't use OpenSSL, so those headers will never exit there.

AFAIK (from my research for that PR) Mac used OpenSSL AND included its headers until Sierra or something. Then they clumsily removed the headers between some subversion of Sierra until they switched to LibreSSL in High Sierra. From the blog posts I found, most developers just used homebrew openssl the during that shitshow.

[asterite]

@LVMBDV Good catch, thank you!

[RX14]

This has all been reverted - we need to come up with a better way of detecting libressl.

[j8r]

Fixed with https://github.com/crystal-lang/crystal/pull/6917

[vinaysolanki]

I first encountered this issue while install Crystal on my Mac and second time while installing icr (Crystal Interactive Shell)

Easiest way I was able to make it work was by doing:

ln -s /usr/local/Cellar/openssl/1.0.2q/lib/libssl.dylib /usr/local/lib/

(Make sure your openssl version is correct)

and then

ln -s /usr/local/opt/openssl/lib/libcrypto.dylib /usr/local/lib

Then continue with installation...

[ndbroadbent]

I also ran into a similar problem while compiling Crystal and running the specs on my Mac (Catalina 10.15.1):

$ make all
Using /usr/local/opt/llvm/bin/llvm-config [version=9.0.0]
c++ -c  -o src/llvm/ext/llvm_ext.o src/llvm/ext/llvm_ext.cc -I/usr/local/Cellar/llvm/9.0.0_1/include -std=c++11 -stdlib=libc++   -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS
cc -fPIC    -c -o src/ext/sigfault.o src/ext/sigfault.c
ar -rcs src/ext/libcrystal.a src/ext/sigfault.o
CRYSTAL_CONFIG_PATH="/Users/ndbroadbent/code/crystal/src" CRYSTAL_CONFIG_LIBRARY_PATH="/usr/local/Cellar/crystal/0.31.1/embedded/lib" CRYSTAL_CONFIG_BUILD_COMMIT="54e68f0b3" ./bin/crystal build  -o .build/crystal src/compiler/crystal.cr -D without_openssl -D without_zlib

$ make spec
Using /usr/local/opt/llvm/bin/llvm-config [version=9.0.0]
./bin/crystal build  --exclude-warnings spec/std --exclude-warnings spec/compiler -o .build/all_spec spec/all_spec.cr
Using compiled compiler at .build/crystal
n
ld: library not found for -lssl (this usually means you need to install the development package for libssl)
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc "${@}" -o '/Users/ndbroadbent/code/crystal/.build/all_spec'  -rdynamic  -lgmp -lxml2 -lyaml -lz `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libssl || printf %s '-lssl -lcrypto'` `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libcrypto || printf %s '-lcrypto'` /Users/ndbroadbent/code/crystal/src/llvm/ext/llvm_ext.o `/usr/local/opt/llvm/bin/llvm-config --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre /usr/local/Cellar/crystal/0.31.1/embedded/lib/libgc.a -lpthread /Users/ndbroadbent/code/crystal/src/ext/libcrystal.a -levent -liconv -ldl -L/usr/local/Cellar/crystal/0.31.1/embedded/lib -L/usr/lib -L/usr/local/lib`
make: *** [.build/all_spec] Error 1

Then I ran these ln commands:

$ ln -s /usr/local/Cellar/openssl@1.1/1.1.1d/lib/libssl.dylib /usr/local/lib/
$ ln -s /usr/local/opt/openssl/lib/libcrypto.dylib /usr/local/lib/

make spec crashed with a different error:

$ make spec
Using /usr/local/opt/llvm/bin/llvm-config [version=9.0.0]
./bin/crystal build  --exclude-warnings spec/std --exclude-warnings spec/compiler -o .build/all_spec spec/all_spec.cr
Using compiled compiler at .build/crystal
Undefined symbols for architecture x86_64:
  "_ERR_load_crypto_strings", referenced from:
      ___crystal_main in _main.o
  "_EVP_MD_CTX_create", referenced from:
      _*OpenSSL::Digest::new_evp_mt_ctx<String>:Pointer(LibCrypto::EVP_MD_CTX_Struct) in O-penS-S-L-5858D-igest.o
      _*OpenSSL::Digest#clone:OpenSSL::Digest in O-penS-S-L-5858D-igest.o
  "_EVP_MD_CTX_destroy", referenced from:
      _*OpenSSL::Digest#finalize:Nil in O-penS-S-L-5858D-igest.o
      _*OpenSSL::Digest#clone:OpenSSL::Digest in O-penS-S-L-5858D-igest.o
  "_OPENSSL_add_all_algorithms_noconf", referenced from:
      ___crystal_main in _main.o
  "_SSL_library_init", referenced from:
      ___crystal_main in _main.o
  "_SSL_load_error_strings", referenced from:
      ___crystal_main in _main.o
  "_SSLv23_method", referenced from:
      _*OpenSSL::SSL::Context::default_method:Pointer(Void) in O-penS-S-L-5858S-S-L-5858C-ontext.o
  "_sk_free", referenced from:
      _~procProc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc2Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc3Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc4Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc5Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc6Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      _~proc7Proc(Pointer(Void), Nil)@src/openssl/ssl/hostname_validation.cr:66 in _main.o
      ...
  "_sk_num", referenced from:
      _*OpenSSL::SSL::HostnameValidation::matches_subject_alternative_name<String, Pointer(Void)>:OpenSSL::SSL::HostnameValidation::Result in O-penS-S-L-5858S-S-L-5858H-ostnameV-alidation.o
  "_sk_pop_free", referenced from:
      _*OpenSSL::SSL::HostnameValidation::matches_subject_alternative_name<String, Pointer(Void)>:OpenSSL::SSL::HostnameValidation::Result in O-penS-S-L-5858S-S-L-5858H-ostnameV-alidation.o
  "_sk_value", referenced from:
      _*OpenSSL::SSL::HostnameValidation::matches_subject_alternative_name<String, Pointer(Void)>:OpenSSL::SSL::HostnameValidation::Result in O-penS-S-L-5858S-S-L-5858H-ostnameV-alidation.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc "${@}" -o '/Users/ndbroadbent/code/crystal/.build/all_spec'  -rdynamic  -lgmp -lxml2 -lyaml -lz `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libssl || printf %s '-lssl -lcrypto'` `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libcrypto || printf %s '-lcrypto'` /Users/ndbroadbent/code/crystal/src/llvm/ext/llvm_ext.o `/usr/local/opt/llvm/bin/llvm-config --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre /usr/local/Cellar/crystal/0.31.1/embedded/lib/libgc.a -lpthread /Users/ndbroadbent/code/crystal/src/ext/libcrystal.a -levent -liconv -ldl -L/usr/local/Cellar/crystal/0.31.1/embedded/lib -L/usr/lib -L/usr/local/lib`
make: *** [.build/all_spec] Error 1

I've installed all these required libraries using brew.

[asterite]

@ndbroadbent In https://crystal-lang.org/install/on_mac_os/ there's an alternative fix that might work. Could you try it out?

brew install openssl
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/opt/openssl/lib/pkgconfig

[ndbroadbent]

I saw that I needed LLVM 8.0, so I downgraded:

$ brew install llvm@8
$ brew unlink llvm
$ brew link --force --overwrite llvm@8
Linking /usr/local/Cellar/llvm@8/8.0.1_1... 474 symlinks created

If you need to have this software first in your PATH instead consider running:
  echo 'export PATH="/usr/local/opt/llvm@8/bin:$PATH"' >> ~/.bash_profile

Then ran that command to add it to my PATH. (But LLVM 8.0 didn't fix the ld: library not found for -lssl error.)

I also removed the symlinks I added earlier:

$ rm /usr/local/lib/libssl.dylib
$ rm /usr/local/lib/libcrypto.dylib

And saw this article about issues with Crystal and openssl. They recommended setting the PKG_CONFIG_PATH variable:

export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

After that, I got this error:

$ make spec
Using /usr/local/opt/llvm@8/bin/llvm-config [version=8.0.1]
./bin/crystal build  --exclude-warnings spec/std --exclude-warnings spec/compiler -o .build/all_spec spec/all_spec.cr
Using compiled compiler at .build/crystal
Undefined symbols for architecture x86_64:
  "llvm::MetadataTracking::track(void*, llvm::Metadata&, llvm::PointerUnion<llvm::MetadataAsValue*, llvm::Metadata*>)", referenced from:
      llvm::MetadataTracking::track(llvm::Metadata*&) in llvm_ext.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Error: execution of command failed with code: 1: `cc "${@}" -o '/Users/ndbroadbent/code/crystal/.build/all_spec'  -rdynamic  -lgmp -lxml2 -lyaml -lz `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libssl || printf %s '-lssl -lcrypto'` `command -v pkg-config > /dev/null && pkg-config --libs --silence-errors libcrypto || printf %s '-lcrypto'` /Users/ndbroadbent/code/crystal/src/llvm/ext/llvm_ext.o `/usr/local/opt/llvm@8/bin/llvm-config --libs --system-libs --ldflags 2> /dev/null` -lstdc++ -lpcre /usr/local/Cellar/crystal/0.31.1/embedded/lib/libgc.a -lpthread /Users/ndbroadbent/code/crystal/src/ext/libcrystal.a -levent -liconv -ldl -L/usr/local/Cellar/crystal/0.31.1/embedded/lib -L/usr/lib -L/usr/local/lib`
make: *** [.build/all_spec] Error 1

I realized that's probably because I changed the LLVM version and/or setting PKG_CONFIG_PATH, so I ran make clean and then make all to rebuild the crystal compiler. Then I was able to get the specs running!

---

I saw that the Circle CI config already has a test_darwin job that tests on MacOS. So I was thinking that it would be great to add a CI job that installs homebrew and the required packages, and makes sure that developers don't run into issues like this. But then I realized that it does already do this! And the only problem is that the docs in the wiki got out of sync with the CI commands.

It would be really nice if there was some way to automate this and use a shared file somewhere, so that the docs + CI are always in sync.

[ndbroadbent]

Thanks @asterite, I also found that PKG_CONFIG_PATH fix in this article!

I installed a fresh copy of macOS last week, and I already had openssl installed, so it would be great if Crystal could compile on a Mac without any errors or troubleshooting steps like this. Would it be possible to detect and fix this issue automatically?

[appcypher]

In addition to the PKG_CONFIG_PATH env variable, I had to brew install pkg-config to get this working. It was not mentioned here but it is in the CI job