Cannot require a .so file in a macro file

[datwaft]

Hello!

I ran into this issue while trying to require, inside a macro file, a LuaRocks module which uses a .so file. The module that I am trying to use is lrexlib, specifically the PCRE2 version.

My objective doing this was to use a feature-complete PCRE2 regex implementation inside my macros. As everything, at the moment, that I tried, worked both on a normal Fennel file and a macro file, I didn't think that this would fail to work.

This issue was already reported in the IRC channel with the following comments from @technomancy, so I opened this issue for tracking purposes.

technomancy@technomancy:libera.chat
when I read your question I was like "there's no way there's a legitimate use
case for loading C code from a macro" ... then I read the issue and saw it was
PCRE and I was like "oh... OK, so I guess there is one"

technomancy@technomancy:libera.chat
short-term I think writing your own macro searcher is the way to go. I will
need to think about this. it seems like overkill to include it out of the box
if there is literally only one single use case for it but if the implementation
is simple maybe

Here is the original issue in the hotpot.nvim repository.

How to reproduce the issue

I created this Github repository which uses Docker to reproduce the issue.

Inside the readme of the repository are the instructions for the execution.

[technomancy]

I couldn't reproduce the exact case in the repo since the library hard-codes a bunch of paths as x86_64 so they don't work on my laptop, but I swapped it out for rex_posix and I think it ends up demonstrating the same underlying problem without triggering the bug in the library.

Anyway I think I found the solution to this problem; you need to install a macro searcher that can use package.cpath and package.loadlib:

local fennel = require("fennel")

table.insert(fennel["macro-searchers"], function(module_name)
   local filename = fennel["search-module"](module_name, package.cpath)
   if filename then
      local func = "luaopen_" .. module_name
      return function() return package.loadlib(filename, func) end, filename
   end
end)

fennel.dofile("test.fnl")

Does that get you what you're looking for?

I think it's probably best to leave this in application code and not add it to Fennel since a macro searcher which can load arbitrary C code would defeat the purpose of the compiler sandbox. But it's important that a workaround exists for people who don't need the compiler sandbox. Maybe this example could go on the wiki or something.

[datwaft]

As can be seen in this branch, it works properly with that solution. Thanks a lot!

Evidence

[datwaft]

Additionally, it also works on Neovim as can be seen in this branch.

[technomancy]

I added docs for this in ec5da6d