search

ocaml-community / ocaml-mariadb

OCaml bindings to MariaDB, supporting the nonblocking API
55 stars 18 forks source link

Installing/building on OSX #4

Closed andreas closed 7 years ago

andreas commented 7 years ago

I'm running OSX 10.11.6 (El Capitan) and OCaml 4.03.0. I've installed MariaDB Connector/C 2.2.2 via Homebrew (brew install mariadb-connector-c).

When trying to install mariadb via opam, I get the following error:

> opam install mariadb
The following actions will be performed:
  ∗  install mariadb 0.5.1

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[mariadb] Archive in cache

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[ERROR] The compilation of mariadb failed at "make".
Processing  1/1: [mariadb: ocamlfind remove]
#=== ERROR while installing mariadb.0.5.1 =====================================#
# opam-version 1.2.2
# os           darwin
# command      make
# path         /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1
# compiler     4.03.0
# exit-code    2
# env-file     /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-86521-58c514.env
# stdout-file  /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-86521-58c514.out
# stderr-file  /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-86521-58c514.err
### stdout ###
# [...]
# /Users/agarnaes/.opam/4.03.0/bin/ocamlfind ocamlc -c -g -annot -bin-annot -I bindings -warn-error +1..45 -warn-error +1..45 -package ctypes.stubs -package ctypes.foreign -I stubgen -I bindings -o stubgen/ffi_stubgen.cmo stubgen/ffi_stubgen.ml
# /Users/agarnaes/.opam/4.03.0/bin/ocamlfind ocamlc -linkpkg -g -warn-error +1..45 -linkpkg -package ctypes.stubs -package ctypes.foreign bindings/mariadb_bindings.cma stubgen/ffi_stubgen.cmo -o stubgen/ffi_stubgen.byte
# stubgen/ffi_stubgen.byte -c > lib/ffi_generated_stubs.c
# /Users/agarnaes/.opam/4.03.0/bin/ocamlfind ocamlc -g -I lib -package ctypes -ccopt -I -ccopt /Users/agarnaes/.opam/4.03.0/lib/ctypes -package unix -package ctypes.stubs -package ctypes.foreign -c lib/ffi_generated_stubs.c
# + /Users/agarnaes/.opam/4.03.0/bin/ocamlfind ocamlc -g -I lib -package ctypes -ccopt -I -ccopt /Users/agarnaes/.opam/4.03.0/lib/ctypes -package unix -package ctypes.stubs -package ctypes.foreign -c lib/ffi_generated_stubs.c
# lib/ffi_generated_stubs.c:1:10: fatal error: 'mysql/mysql.h' file not found
# #include <mysql/mysql.h>
#          ^
# 1 error generated.
# Command exited with code 2.
### stderr ###
# W: Cannot find source file matching module 'Ffi_generated' in library mariadb
# W: Cannot find source file matching module 'Ffi_generated_types' in library mariadb
# E: Failure("Command ''/Users/agarnaes/.opam/4.03.0/bin/ocamlbuild' bindings/mariadb_bindings.cma bindings/mariadb_bindings.cmxa bindings/mariadb_bindings.a bindings/mariadb_bindings.cmxs lib/libmariadb_stubs.a lib/dllmariadb_stubs.so lib/mariadb.cma lib/mariadb.cmxa lib/mariadb.a lib/mariadb.cmxs stubgen/ffi_stubgen.byte stubgen/ffi_types_stubgen.byte examples/select/nonblocking_select_example.native examples/blocking/blocking_example.native -use-ocamlfind -tag debug' terminated with error code 10")
# make: *** [build] Error 1

It seems like mysql.h is under mariadb/mysql.h in my setup. If I create a symlink to remedy this, I get the following error instead:

> opam install mariadb
The following actions will be performed:
  ∗  install mariadb 0.5.1

=-=- Gathering sources =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[mariadb] Archive in cache

=-=- Processing actions -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=  🐫
[ERROR] The compilation of mariadb failed at "make".
Processing  1/1: [mariadb: ocamlfind remove]
#=== ERROR while installing mariadb.0.5.1 =====================================#
# opam-version 1.2.2
# os           darwin
# command      make
# path         /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1
# compiler     4.03.0
# exit-code    2
# env-file     /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-87151-58c514.env
# stdout-file  /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-87151-58c514.out
# stderr-file  /Users/agarnaes/.opam/4.03.0/build/mariadb.0.5.1/mariadb-87151-58c514.err
### stdout ###
# stubgen/ffi_ml_types_stubgen.c:466:71: error: use of undeclared identifier 'MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS'
# [...]
# stubgen/ffi_ml_types_stubgen.c:467:15: error: use of undeclared identifier 'MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS'; did you mean 'check_MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS_const'?
#      int v = (MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS);
#               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#               check_MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS_const
# stubgen/ffi_ml_types_stubgen.c:466:13: note: 'check_MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS_const' declared here
#      enum { check_MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS_const = (int)MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS };
#             ^
# 6 errors generated.
# Command exited with code 1.
### stderr ###
# W: Cannot find source file matching module 'Ffi_generated' in library mariadb
# W: Cannot find source file matching module 'Ffi_generated_types' in library mariadb
# E: Failure("Command ''/Users/agarnaes/.opam/4.03.0/bin/ocamlbuild' bindings/mariadb_bindings.cma bindings/mariadb_bindings.cmxa bindings/mariadb_bindings.a bindings/mariadb_bindings.cmxs lib/libmariadb_stubs.a lib/dllmariadb_stubs.so lib/mariadb.cma lib/mariadb.cmxa lib/mariadb.a lib/mariadb.cmxs stubgen/ffi_stubgen.byte stubgen/ffi_types_stubgen.byte examples/select/nonblocking_select_example.native examples/blocking/blocking_example.native -use-ocamlfind -tag debug' terminated with error code 10")
# make: *** [build] Error 1

i.e. the constant MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS is not defined. To investigate this further, I checked out the repo and looked into this missing constant. As far as I can tell, the constants MYSQL_OPT_CAN_HANDLE_EXPIRED_PASSWORDS and MYSQL_OPT_USE_THREAD_SPECIFIC_MEMORY are not defined in any header files in MariaDB Connector/C 2.2.2, nor in any other version I can find on the releases homepage.

So to make it build, I tried removing use of those constants (and also changed the mysql/mysql.h to mariadb/mysql.h): https://gist.github.com/andreas/0e580b19df542f1dbe1ee2676a2c6768

With the above patch, I got a little further, but sadly still an error when trying to build:

> ocaml setup.ml -build
W: Cannot find source file matching module 'Ffi_generated' in library mariadb
W: Cannot find source file matching module 'Ffi_generated_types' in library mariadb
Finished, 0 targets (0 cached) in 00:00:00.
+ /Users/agarnaes/.opam/4.03.0/bin/ocamlfind ocamlopt -linkpkg -g -I lib -warn-error +1..45 -linkpkg -package unix -package ctypes.stubs -package ctypes.foreign bindings/mariadb_bindings.cmxa lib/mariadb.cmxa examples/select/nonblocking_select_example.cmx -o examples/select/nonblocking_select_example.native
Undefined symbols for architecture x86_64:
  "_ASN1_STRING_data", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_ASN1_STRING_length", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_CONF_modules_free", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_CONF_modules_unload", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_THREADID_set_callback", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_THREADID_set_numeric", referenced from:
      _my_cb_threadid in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_cleanup_all_ex_data", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_num_locks", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_set_id_callback", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_CRYPTO_set_locking_callback", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_ERR_free_strings", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_ERR_get_error", referenced from:
      _my_SSL_error in libmysqlclient.a(ma_secure.c.o)
  "_ERR_reason_error_string", referenced from:
      _my_SSL_error in libmysqlclient.a(ma_secure.c.o)
  "_ERR_remove_state", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
      _my_thread_end in libmysqlclient.a(my_thr_init.c.o)
  "_EVP_MD_size", referenced from:
      _ma_ssl_verify_fingerprint in libmysqlclient.a(ma_secure.c.o)
  "_EVP_cleanup", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_EVP_sha1", referenced from:
      _ma_ssl_verify_fingerprint in libmysqlclient.a(ma_secure.c.o)
  "_OPENSSL_add_all_algorithms_noconf", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_OPENSSL_config", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CIPHER_get_name", referenced from:
      _mysql_get_ssl_cipher in libmysqlclient.a(libmariadb.c.o)
  "_SSL_COMP_get_compression_methods", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_check_private_key", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_free", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_get_cert_store", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_load_verify_locations", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_new", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_set_cipher_list", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_set_default_verify_paths", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_set_verify", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_set_verify_depth", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_use_PrivateKey_file", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_CTX_use_certificate_file", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_SESSION_set_timeout", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_clear", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_connect", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_free", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_close in libmysqlclient.a(ma_secure.c.o)
      _client_mpvio_write_packet in libmysqlclient.a(my_auth.c.o)
  "_SSL_get_current_cipher", referenced from:
      _mysql_get_ssl_cipher in libmysqlclient.a(libmariadb.c.o)
  "_SSL_get_error", referenced from:
      _my_ssl_async_check_result in libmysqlclient.a(mysql_async.c.o)
  "_SSL_get_ex_data", referenced from:
      _my_verify_callback in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
      _ma_ssl_verify_fingerprint in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_SSL_get_ex_data_X509_STORE_CTX_idx", referenced from:
      _my_verify_callback in libmysqlclient.a(ma_secure.c.o)
  "_SSL_get_peer_certificate", referenced from:
      _ma_ssl_verify_fingerprint in libmysqlclient.a(ma_secure.c.o)
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_SSL_get_session", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_get_verify_result", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_library_init", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_SSL_load_error_strings", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_SSL_new", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_read", referenced from:
      _my_ssl_read_async in libmysqlclient.a(mysql_async.c.o)
      _my_ssl_read in libmysqlclient.a(ma_secure.c.o)
  "_SSL_set_ex_data", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_SSL_set_fd", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_SSL_set_quiet_shutdown", referenced from:
      _my_ssl_close in libmysqlclient.a(ma_secure.c.o)
  "_SSL_shutdown", referenced from:
      _my_ssl_close in libmysqlclient.a(ma_secure.c.o)
  "_SSL_write", referenced from:
      _my_ssl_write_async in libmysqlclient.a(mysql_async.c.o)
      _my_ssl_write in libmysqlclient.a(ma_secure.c.o)
  "_TLSv1_client_method", referenced from:
      _my_ssl_start in libmysqlclient.a(ma_secure.c.o)
  "_X509_NAME_ENTRY_get_data", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_X509_NAME_get_entry", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_X509_NAME_get_index_by_NID", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_X509_STORE_CTX_get_current_cert", referenced from:
      _my_verify_callback in libmysqlclient.a(ma_secure.c.o)
  "_X509_STORE_CTX_get_error_depth", referenced from:
      _my_verify_callback in libmysqlclient.a(ma_secure.c.o)
  "_X509_STORE_CTX_get_ex_data", referenced from:
      _my_verify_callback in libmysqlclient.a(ma_secure.c.o)
  "_X509_STORE_load_locations", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_X509_STORE_set_flags", referenced from:
      _my_ssl_init in libmysqlclient.a(ma_secure.c.o)
  "_X509_digest", referenced from:
      _ma_ssl_verify_fingerprint in libmysqlclient.a(ma_secure.c.o)
  "_X509_free", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_X509_get_subject_name", referenced from:
      _my_ssl_verify_server_cert in libmysqlclient.a(ma_secure.c.o)
  "_X509_verify_cert_error_string", referenced from:
      _my_ssl_connect in libmysqlclient.a(ma_secure.c.o)
  "_iconv", referenced from:
      _mariadb_convert_string in libmysqlclient.a(my_charset.c.o)
     (maybe you meant: _camlCamlinternalFormat__format_of_iconv_62392, _camlCamlinternalFormat__bprint_iconv_flag_1425 , _camlCamlinternalFormat__char_of_iconv_1354 , _camlCamlinternalFormat__format_of_iconvL_62394 , _camlCamlinternalFormat__format_of_iconvn_62398 , _camlCamlinternalFormat__format_of_iconvl_62396 )
  "_iconv_close", referenced from:
      _mariadb_convert_string in libmysqlclient.a(my_charset.c.o)
  "_iconv_open", referenced from:
      _mariadb_convert_string in libmysqlclient.a(my_charset.c.o)
  "_sk_free", referenced from:
      _my_ssl_end in libmysqlclient.a(ma_secure.c.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
File "caml_startup", line 1:
Error: Error during linking
Command exited with code 2.
Compilation unsuccessful after building 68 targets (65 cached) in 00:00:01.
E: Failure("Command ''/Users/agarnaes/.opam/4.03.0/bin/ocamlbuild' bindings/mariadb_bindings.cma bindings/mariadb_bindings.cmxa bindings/mariadb_bindings.a bindings/mariadb_bindings.cmxs lib/libmariadb_stubs.a lib/dllmariadb_stubs.so lib/mariadb.cma lib/mariadb.cmxa lib/mariadb.a lib/mariadb.cmxs stubgen/ffi_stubgen.byte stubgen/ffi_types_stubgen.byte examples/select/nonblocking_select_example.native examples/blocking/blocking_example.native -use-ocamlfind -tag debug' terminated with error code 10")

It seems the problem is that brew puts .dylib's in /usr/local/lib/mariadb/. I was able to fix it by symlinking libmysqlclient.dylib, libmariadb.dylib and libmariadb.2.dylib into /usr/local/lib.

As such, I got it working in the end, but it took quite some hairpulling 😄 It would be great if the library installed cleanly via opam on OSX.

andrenth commented 7 years ago

The whole MariaDB client library situation is a bit confusing right now... If you install MariaDB (up to 10.1) you get a libmariadbclient that is compatible with MySQL's libmysqlclient, so you get the include files in /usr/include/mysql for example. If you install their C Connector, you end up with /usr/include/mariadb. The same is true for those constants, which exist in libmariadbclient but not in the C connector.

To further complicate things, it appears that starting with MariaDB 10.2, libmariadbclient will be obsolete, and using the C connector will be the supported way to do things. That's fine by me, but for some reason there's no C connector package (at least for Ubuntu) in their own repositories, unless I missed something.

Anyway, the correct way to fix all these issues would be to add some sort of preprocessing to try detect what client library is installed and use the correct paths. This seems too complicated though, and since the C Connector will be the right way to do things going forward, I'd be willing to only support that in the future. But it would be nice if MariaDB would at least add it to their repositories...

andrenth commented 7 years ago

Oh, w.r.t /usr/local/lib/mariadb, shouldn't brew have updated your /etc/ld.so.conf or equivalent in OSX so that it's added to the library search path?

andreas commented 7 years ago

That does indeed sound confusing.

As far as I can tell, /usr/local/lib/mariadb is not added automatically to the library search path by brew. I guess that you have to do that manually when linking.

I should also add: thanks a lot for putting together the library! It's great with a Lwt-enabled MySQL client (without Lwt_preemptive).

andrenth commented 7 years ago

Well, I've found a PPA for version 2.3.1 of the connector, and opened a request for them to backport the newer version to Ubuntu 16.04 (which only has version 2.0.0, which doesn't have the nonblocking functions).

So I guess I'll work on adapting the library to use the connector instead of libmariadbclient, since that'll be the future anyway. At least it should solve the mysql.h issue, and also get rid of those constants, so the installation process will be a bit easier for you.

andrenth commented 7 years ago

I've removed the bindings to functions and flags that are absent in Connector/C, and changed the build infrastructure to detect dynamically if it should link to the connector's libmariadb or to libmysqlclient. Preference is given to the connector as it seems libmysqlclient will be deprecated starting in MariaDB 10.2.

Other than the /usr/local/lib/mariadb issue, you should be able to keep your hair while installing the library :)

Please reopen if you find any problems.

andreas commented 7 years ago

You work fast 😄 It works great. I can install 0.7.0 cleanly using opam (with the .dylib symlinks in place). Thanks!

andrenth commented 7 years ago

Awesome :smiley: