Support libmysqlclient.22.dylib in indexer

sanikolaev commented 9 months ago

In macOS Ventura (and probably earlier versions) the brew package mysql provides libmysqlclient.22.dylib:

➜  ~ brew info mysql
==> mysql: stable 8.2.0 (bottled)

➜  ~ brew ls mysql|grep libmysqlclient
/opt/homebrew/Cellar/mysql/8.1.0/lib/libmysqlclient.22.dylib
/opt/homebrew/Cellar/mysql/8.1.0/lib/libmysqlclient.dylib
/opt/homebrew/Cellar/mysql/8.1.0/lib/libmysqlclient.a
➜  ~

while indexer wants libmysqlclient.21.dylib

➜  ~ indexer -v
Manticore 6.2.13 9413a5374@24011222 dev (columnar 2.2.5 1d1e432@231204) (secondary 2.2.5 1d1e432@231204) (knn 2.2.5 1d1e432@231204)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

➜  ~ indexer
Manticore 6.2.13 9413a5374@24011222 dev (columnar 2.2.5 1d1e432@231204) (secondary 2.2.5 1d1e432@231204) (knn 2.2.5 1d1e432@231204)
Copyright (c) 2001-2016, Andrew Aksyonoff
Copyright (c) 2008-2016, Sphinx Technologies Inc (http://sphinxsearch.com)
Copyright (c) 2017-2023, Manticore Software LTD (https://manticoresearch.com)

Built on Linux x86_64 for Darwin arm64 (macos) by Clang 15.0.7 compiler.

Configured with these definitions: -DDISTR_BUILD=macos -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=/opt/homebrew/opt/zstd/lib/libzstd.1.dylib -DWITH_CURL=1 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=/opt/homebrew/opt/unixodbc/lib/libodbc.2.dylib -DWITH_EXPAT=1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=/opt/homebrew/opt/mysql-client/lib/libmysqlclient.21.dylib -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=/opt/homebrew/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib -DLOCALDATADIR=/opt/homebrew/var/manticore -DFULL_SHARE_DIR=/opt/homebrew/share/manticore

If I run indexer with libmysqlclient.22.dylib, it works passing most of the tests (the rest likely fail due to another issue).

The task is to make indexer use libmysqlclient.22.dylib when libmysqlclient.21.dylib is not available. Note, the user doens't have to do anything for that (like using a special environment variable or smth).

sanikolaev commented 5 months ago

There are the env. vars.

MYSQL_LIB
POSTGRESQL_LIB

that can be used to specify a path to the corresponding library. Let's test various versions of mysql/mariadb libraries/connectors and postgres versions) in various operating systems and for for each case (different mysql/mariadb/postgresql lib) let's make sure indexer works in:

simple mode (we should have a basic CLT test about it)
and for the case of mysql - with ssl (https://manual.manticoresearch.com/Data_creation_and_modification/Adding_data_from_external_storages/Fetching_from_databases/Database_connection#SSL-certificate-settings)

Once we have a table of:

operating system
supported mysql/mariadb client versions
supported postgresql lib versions

We'll decide how we can change the indexer behaviour in terms of preloading this or that library version automatically.

cyformatician commented 3 months ago

@sanikolaev

Facing a somewhat similar issue, after creating a symlink libmysqlclient.21.dylib -> libmysqlclient.23.dylib in /usr/local/opt/mysql-client/lib/ under macOS Monterey 12.7.6

It appears indexer is not compatible with mysql-client 8.3.x ... ver 8.2.x works fine without crashing.

*** Oops, indexer crashed! Please send the following report to developers. Manticore 6.3.2 c296dc7c8@24062606 (columnar 2.3.0 88a01c3@24052206) (secondary 2.3.0 88a01c3@24052206) (knn 2.3.0 88a01c3@24052206) -------------- report begins here --------------- Current document: docid=0, hits=0 Current batch: minid=0, maxid=0 Hit pool start: docid=0, hit=0 -------------- backtrace begins here --------------- Program compiled with Clang 16.0.6 Configured with flags: Configured with these definitions: -DDISTR_BUILD=macos -DUSE_SYSLOG=1 -DWITH_GALERA=1 -DWITH_RE2=1 -DWITH_RE2_FORCE_STATIC=1 -DWITH_STEMMER=1 -DWITH_STEMMER_FORCE_STATIC=1 -DWITH_NLJSON=1 -DWITH_UNIALGO=1 -DWITH_ICU=1 -DWITH_ICU_FORCE_STATIC=1 -DWITH_SSL=1 -DWITH_ZLIB=1 -DWITH_ZSTD=1 -DDL_ZSTD=1 -DZSTD_LIB=/usr/local/opt/zstd/lib/libzstd.1.dylib -DWITH_CURL=1 -DWITH_ODBC=1 -DDL_ODBC=1 -DODBC_LIB=/usr/local/opt/unixodbc/lib/libodbc.2.dylib -DWITH_EXPAT=1 -DWITH_ICONV=1 -DWITH_MYSQL=1 -DDL_MYSQL=1 -DMYSQL_LIB=/usr/local/opt/mysql-client/lib/libmysqlclient.21.dylib -DWITH_POSTGRESQL=1 -DDL_POSTGRESQL=1 -DPOSTGRESQL_LIB=/usr/local/opt/postgresql@14/lib/postgresql@14/libpq.5.dylib -DLOCALDATADIR=/usr/local/var/manticore -DFULL_SHARE_DIR=/usr/local/share/manticore

Attaching crash report.

crash-report-mysqlclient8.3.txt

cyformatician commented 2 months ago

Quick follow-up: The latest Manticore release (6.2.12) on macOS installs MySQL client v9, which appears to be incompatible with MariaDB. While ManticoreSearch can connect to MariaDB, the indexer can't, as it complains about a missing MySQL authentication plugin:

ERROR: table 'XXXXXX': sql_connect: Authentication plugin 'mysql_native_password' cannot be loaded: dlopen(/usr/local/Cellar/mysql-client/9.0.1/lib/plugin/mysql_native_password.so, 0x0002): tried: '/usr/local/Cellar/mysql-client/9.0.1/lib/plugin/mysql_native_password.so' (no such file)

MySQL client v9 dropped support for mysql_native_password, and MariaDB doesn't support MySQL client v9's authentication protocols - https://dev.mysql.com/doc/refman/9.0/en/pluggable-authentication.html#pluggable-authentication-available-plugins

manticoresoftware / manticoresearch

Support libmysqlclient.22.dylib in indexer #1732