Trying to get exqlite to work on Darwin (x86_64)

[clarkcb]

Hi, I'm trying to add sqlite3 support to an Elixir CLI tool I built recently, and exqlite seems like a good match, but I'm unable to get it working. The main error seems to be this:

** (UndefinedFunctionError) function Exqlite.Sqlite3NIF.open/2 is undefined (module Exqlite.Sqlite3NIF is not available)

I was able to determine that the NIF part apparently failed to compile, and after some tinkering I was seemingly able to get it to compile. I modified the Makefile to add this for Darwin (added the LIBNAME line to change extension to dylib):

ifeq ($(KERNEL_NAME), Darwin)
    CFLAGS += -fPIC
    LDFLAGS += -dynamiclib -undefined dynamic_lookup
    LIB_NAME = $(PREFIX)/sqlite3_nif.dylib
endif

I also exported some ENV vars (both erlang and elixir are installed via homebrew):

MIX_APP_PATH=/path/to/deps/exqlite
ERL_EI_INCLUDE_DIR=/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include
ERTS_INCLUDE_DIR=/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include
V=1

I had to add both of the include dirs to get the compilation to work. Not sure where ERTS_INCLUDE_DIR is supposed to get defined normally, but if I don't define it I get this output:

$ make all  
 CC sqlite3_nif.o
cc -c -I"/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include" -I"" -Ic_src -DNDEBUG=1 -O2 -fPIC -DSQLITE_THREADSAFE=1 -DSQLITE_USE_URI=1 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 -DSQLITE_DQS=0 -DHAVE_USLEEP=1 -DALLOW_COVERING_INDEX_SCAN=1 -DENABLE_FTS3_PARENTHESIS=1 -DENABLE_LOAD_EXTENSION=1 -DENABLE_SOUNDEX=1 -DENABLE_STAT4=1 -DENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1 -DSQLITE_ENABLE_RBU=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_OMIT_DEPRECATED=1 -DSQLITE_ENABLE_DBSTAT_VTAB=1 -o /path/to/deps/exqlite/obj/sqlite3_nif.o c_src/sqlite3_nif.c
c_src/sqlite3_nif.c:1003:10: error: call to undeclared function 'sqlite3_enable_load_extension'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
    rc = sqlite3_enable_load_extension(conn->db, enable_load_extension_value);
         ^
c_src/sqlite3_nif.c:1003:10: note: did you mean 'exqlite_enable_load_extension'?
c_src/sqlite3_nif.c:984:1: note: 'exqlite_enable_load_extension' declared here
exqlite_enable_load_extension(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[])
^
1 error generated.
make: *** [/path/to/deps/exqlite/obj/sqlite3_nif.o] Error 1

With both ERL_EI_INCLUDE_DIR and ERTS_INCLUDE_DIR defined, I get this output:

$ make all                                                                              
 CC sqlite3_nif.o
cc -c -I"/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include" -I"/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include" -Ic_src -DNDEBUG=1 -O2 -fPIC -DSQLITE_THREADSAFE=1 -DSQLITE_USE_URI=1 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 -DSQLITE_DQS=0 -DHAVE_USLEEP=1 -DALLOW_COVERING_INDEX_SCAN=1 -DENABLE_FTS3_PARENTHESIS=1 -DENABLE_LOAD_EXTENSION=1 -DENABLE_SOUNDEX=1 -DENABLE_STAT4=1 -DENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1 -DSQLITE_ENABLE_RBU=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_OMIT_DEPRECATED=1 -DSQLITE_ENABLE_DBSTAT_VTAB=1 -o /path/to/deps/exqlite/obj/sqlite3_nif.o c_src/sqlite3_nif.c
 CC sqlite3.o
cc -c -I"/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include" -I"/usr/local/Cellar/erlang/26.2.5/lib/erlang/erts-14.2.5/include" -Ic_src -DNDEBUG=1 -O2 -fPIC -DSQLITE_THREADSAFE=1 -DSQLITE_USE_URI=1 -DSQLITE_LIKE_DOESNT_MATCH_BLOBS=1 -DSQLITE_DQS=0 -DHAVE_USLEEP=1 -DALLOW_COVERING_INDEX_SCAN=1 -DENABLE_FTS3_PARENTHESIS=1 -DENABLE_LOAD_EXTENSION=1 -DENABLE_SOUNDEX=1 -DENABLE_STAT4=1 -DENABLE_UPDATE_DELETE_LIMIT=1 -DSQLITE_ENABLE_FTS3=1 -DSQLITE_ENABLE_FTS4=1 -DSQLITE_ENABLE_FTS5=1 -DSQLITE_ENABLE_GEOPOLY=1 -DSQLITE_ENABLE_MATH_FUNCTIONS=1 -DSQLITE_ENABLE_RBU=1 -DSQLITE_ENABLE_RTREE=1 -DSQLITE_OMIT_DEPRECATED=1 -DSQLITE_ENABLE_DBSTAT_VTAB=1 -o /path/to/deps/exqlite/obj/sqlite3.o c_src/sqlite3.c
 LD sqlite3_nif.dylib
cc -o /path/to/deps/exqlite/priv/sqlite3_nif.dylib /path/to/deps/exqlite/obj/sqlite3_nif.o /path/to/deps/exqlite/obj/sqlite3.o -dynamiclib -undefined dynamic_lookup

I see the compiled files under obj and priv, so I assume the build is successful. Next I ran mix deps.compile exqlite from the project root and see no error output, and I run mix escript.build to build the tool. But I'm still getting the same error when I try to run it:

$ ./bin/exfind

15:50:48.782 [warning] The on_load function for module Elixir.Exqlite.Sqlite3NIF returned:
{:function_clause,
 [
   {:filename, :join, [{:error, :bad_name}, ~c"sqlite3_nif"],
    [file: ~c"filename.erl", line: 454]},
   {Exqlite.Sqlite3NIF, :load_nif, 0,
    [file: ~c"lib/exqlite/sqlite3_nif.ex", line: 16]},
   {:code_server, :"-handle_on_load/5-fun-0-", 1,
    [file: ~c"code_server.erl", ...]}
 ]}

15:50:48.782 [error] Process #PID<0.114.0> raised an exception
** (FunctionClauseError) no function clause matching in :filename.join/2
    (stdlib 5.2.3) filename.erl:454: :filename.join({:error, :bad_name}, ~c"sqlite3_nif")
    (exqlite 0.23.0) lib/exqlite/sqlite3_nif.ex:16: Exqlite.Sqlite3NIF.load_nif/0
    (kernel 9.2.4) code_server.erl:1393: anonymous fn/1 in :code_server.handle_on_load/5

15:50:48.818 [error] Process #PID<0.116.0> raised an exception
** (FunctionClauseError) no function clause matching in :filename.join/2
    (stdlib 5.2.3) filename.erl:454: :filename.join({:error, :bad_name}, ~c"sqlite3_nif")
    (exqlite 0.23.0) lib/exqlite/sqlite3_nif.ex:16: Exqlite.Sqlite3NIF.load_nif/0
    (kernel 9.2.4) code_server.erl:1393: anonymous fn/1 in :code_server.handle_on_load/5

15:50:48.819 [warning] The on_load function for module Elixir.Exqlite.Sqlite3NIF returned:
{:function_clause,
 [
   {:filename, :join, [{:error, :bad_name}, ~c"sqlite3_nif"],
    [file: ~c"filename.erl", line: 454]},
   {Exqlite.Sqlite3NIF, :load_nif, 0,
    [file: ~c"lib/exqlite/sqlite3_nif.ex", line: 16]},
   {:code_server, :"-handle_on_load/5-fun-0-", 1,
    [file: ~c"code_server.erl", ...]}
 ]}

** (UndefinedFunctionError) function Exqlite.Sqlite3NIF.open/2 is undefined (module Exqlite.Sqlite3NIF is not available)
    (exqlite 0.23.0) Exqlite.Sqlite3NIF.open(~c"/path/to/sqlite.db", 6)
    (exfind 0.1.0) lib/filetypes.ex:59: ExFind.FileTypes.new/0
    (exfind 0.1.0) lib/exfind.ex:56: ExFind.Main.find/2
    (exfind 0.1.0) lib/exfind.ex:69: ExFind.Main.main/1
    (elixir 1.17.2) lib/kernel/cli.ex:136: anonymous fn/3 in Kernel.CLI.exec_fun/2

I tried taking a look at sqlite3_nif.ex to see if anything obvious jumped out, but nothing caught my eye. Unfortunately, I'm somewhat new to elixir and very new to NIF, so I'm not sure where to go from here. Any guidance you might have would be appreciated. Thanks!

[warmwaffles]

This actually just made me realize that I only have CI running for windows and ubuntu, but precompile works fine for mac. 😬

[warmwaffles]

https://github.com/elixir-sqlite/exqlite/blob/fec606ccc8a291526d1e3137099b6acf0f6f84fe/.github/workflows/precompile.yml#L20-L24

https://github.com/elixir-sqlite/exqlite/blob/fec606ccc8a291526d1e3137099b6acf0f6f84fe/.github/workflows/ci.yml#L50-L56

[warmwaffles]

Out of curiosity, is the precompiled package not working for you @clarkcb?

I assume it's here? https://github.com/clarkcb/xfind/tree/main/elixir/exfind can you push your changes up to a feature branch so that I can check it out? I've got access to a mac book air that I can fire this up on.

[clarkcb]

Thanks @warmwaffles, I can do that. I did download and try the latest precompiled .so file, I still got the same error. I don't know whether the name of the library file is required to end in .dylib on macos (I always assumed it was just a convention), but I tried the file with .so extension and also renamed to .dylib, same results for both. By they way, I should mention that I just dropped the file into the deps/exqlite/priv folder, don't know if there was more I needed to do or not.

[clarkcb]

Hi @warmwaffles , I pushed a branch called use-sqlite up to xfind repo. I haven't done any of the db querying implementation in elixir yet, so far I'm just trying to open a new db connection, which you can see in elixir/exfind/lib/filetypes.ex on line 43. The path to the db file is defined as Exfind.Config.xfind_db_path in config.ex. You can either set an env variable called XFIND_PATH with the path that xfind was cloned to, or you can just hard-code a path to any sqlite db, since the task is just to succeed in opening a db connection. After running mix dep.get and mix escript.build and then trying to run bin/exfind, you should see the error shown above. Thanks in advance for looking at this!

[warmwaffles]

This appears to compile fine in Linux 😞

You shouldn't need to copy anything from the precompiled stuff. That gets pulled in automatically when you do mix compile. It should pull the correct dylib.

[clarkcb]

Ok, so I did a little more digging, and it appears the failure actually happens on line 16 of _sqlite3nif.ex:

    path = :filename.join(:code.priv_dir(:exqlite), ~c"sqlite3_nif")

If I replace the :code.priv_dir call with a hard-coded directory, I get slightly farther, I then end up with a message saying it can't find the .so file (since it's currently named with .dylib). I tried renaming the file to have a .so extension and it now seems to work!

So I haven't had a chance to dig into the erlang code module to see what priv_dir is doing, but it seems like maybe that module has some os-specific quirks? I'll explore more when I have a chance, but for now I at least have a workaround. Thanks again.

[clarkcb]

A quick follow-up: :code.priv_dir(:exqlite) is returning {'error', 'bad_name'}. I searched for that in github.com/erlang/otp/blob/master/lib/kernel/src/code.erl and found this note for the lib_dir function that seems relevant:

Returns {error, bad_name} if Name is not the name of an application under $OTPROOT/lib or on a directory referred to through environment variable ERL_LIBS. Fails with an exception if Name has the wrong type.

I tried setting ERL_LIBS to /path/to/deps/exqlite, but that didn't seem to make a difference. Looking at the Erlang kernel code documentation, I'm wondering if maybe the Erlang code server process isn't running in the right mode (interactive?) on the Mac to allow for registering / loading external modules.

[warmwaffles]

How is Erlang and Elixir installed on your machine? Are you using asdf or mise?

[clarkcb]

I installed Erlang and Elixir on the Mac via Homebrew. I've looked at their "formulae" (install scripts) and nothing really jumps out as a possible cause. That said, I only really needed to run on it on the Mac during development and initial testing, so I'm satisfied with the workaround and the fact that it'll run fine on Linux. Btw, I was not familiar with asdf or mise, but they look interesting. Thanks!

[warmwaffles]

No problem. Personally I use mise. I used to use asdf but I noticed it slowing my shell down considerably and any program boots. Your mileage may vary here.

As for asdf I implemented it for the precompile step in the workflows:

https://github.com/elixir-sqlite/exqlite/blob/3b8c7903231cd69ea15e33764efae4fdf9b78b2e/.github/workflows/macos-precompile.yml#L46-L51

I definitely recommend using either one. It's helpful for managing versions of pythons, nodes, elixirs, and erlangs to name a short few.