attempt to yield across metamethod/C-call boundary

[ltratt]

Description

On neovim-0.5 on OpenBSD (which uses Lua, not LuaJIT, to run things), a97af306c4e9c9a6fa7c886c0ffe3079822c5203 (found with git bisect) causes Telescope find_files to stop working, though in two different ways depending on my config.

On my "main" config, as soon as I call Telescope find_files I get:

(tail call):-1: Finder failed with msg:  attempt to yield across metamethod/C-call boundary

On a minimal config, in contrast, I don't get this error message but no files are listed at all.

Here's the complete nvim --version:

$ nvim --version                                               ~
NVIM v0.5.0
Build type: Release
Lua 5.1
Compilation: /home/ports/pobj/neovim-0.5.0/bin/cc -O2 -pipe -g -DNDEBUG -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/home/ports/pobj/neovim-0.5.0/build-amd64/config -I/home/ports/pobj/neovim-0.5.0/neovim-0.5.0/src -I/usr/local/include -I/home/ports/pobj/neovim-0.5.0/build-amd64/deps/include -I/home/ports/pobj/neovim-0.5.0/build-amd64/src/nvim/auto -I/home/ports/pobj/neovim-0.5.0/build-amd64/include
Compiled by _pbuild@overdrive.tratt.net

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/local/share/nvim"

Run :checkhealth for more info

Neovim version

NVIM v0.5.0

Operating system and version

OpenBSD

Steps to reproduce

Telescope find_files

Expected behavior

No response

Actual behavior

(tail call):-1: Finder failed with msg:  attempt to yield across metamethod/C-call boundary

Minimal config

call plug#begin('~/.config/nvim/plugged')
Plug 'nvim-lua/plenary.nvim'
Plug 'nvim-telescope/telescope.nvim'
call plug#end()

[tjdevries]

Hmm, but in both cases nvim is being used with lua and not luajit?

My guess is that the problem is the metamethod boundary, which I didn't know Lua couldn't yield across (the C-boundary one is shared between LuaJIT and lua, as far as I know).

[ltratt]

Hmm, but in both cases nvim is being used with lua and not luajit?

Yes in both cases with Lua (with LuaFFI from https://github.com/facebookarchive/luaffifb).

[tjdevries]

Oh interesting! You're using the ffi package from there installed and accessible from within nvim?

[ltratt]

You're using the ffi package from there installed and accessible from within nvim?

I am indeed! I ported LuaFFI to OpenBSD mostly for Telescope! https://marc.info/?l=openbsd-ports&m=162981402825140&w=2

[tjdevries]

This happens every time you call find files? Small and large directories?

I hopefully have time to recompile nvim with Lua only this week and find a solution.

[ltratt]

This happens every time you call find files? Small and large directories?

Yes and yes respectively.

I hopefully have time to recompile nvim with Lua only this week and find a solution.

Thanks -- and sorry for being a pain!

[ltratt]

I've had a very small dig into this, and I think the offending yield might be in plenary's M.wrap function. But, there again, the only obvious places calling wrap in telescope seem to be test files, so perhaps it's another yield elsewhere. I'm afraid that I'm not really sure how to debug Lua in neovim other than placing print statements in various places and praying!

[tjdevries]

So I'm pretty sure the problem is that finder is using __call metamethod and that we yield from _inside of the __call_. So if I perhaps change the signature to be find finder.find(...) instead of finder(...) then we may be able to avoid the problem.

Sorry, this week ended up being busier than anticipated and I didn't have time to check it out on Friday.

[ltratt]

I guess you're referring to this bit:

  obj.__call = function(t, ...)
    return t:_find(...)
  end

at line 16 in lua/telescope/finders.lua ? I'm happy to try out any suggested changes if you have them!

[tjdevries]

I'm hoping to do a bit of work on telescope tomorrow -- if I don't get a chance to test it out then, ping me again in here and I'll write out some steps on how I'm guessing we can fix it :)

[ltratt]

@tjdevries You asked for a ping, so as requested, I'm pinging :) If you can give me some clues, I'm happy to see if I can get anywhere with this.

[Nis5l]

Any progress or idea on how i could fix this?

For me the results portion for find_files and live_grep is blank and i get the same warning.

nvim --version

NVIM v0.5.1
Build type: Gentoo
Lua 5.1
Compilation: /usr/bin/x86_64-pc-linux-gnu-gcc -march=skylake -O2 -pipe -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET_MATCH_LIMIT -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/var/tmp/portage/app-editors/neovim-0.5.1/work/neovim-0.5.1_build/config -I/var/tmp/portage/app-editors/neovim-0.5.1/work/neovim-0.5.1/src -I/usr/include -I/var/tmp/portage/app-editors/neovim-0.5.1/work/neovim-0.5.1_build/src/nvim/auto -I/var/tmp/portage/app-editors/neovim-0.5.1/work/neovim-0.5.1_build/include
Compiled by portage@localhost

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "/etc/vim/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

[tjdevries]

Do you have ripgrep installed? @Nis5l

[tjdevries]

@tjdevries You asked for a ping, so as requested, I'm pinging :) If you can give me some clues, I'm happy to see if I can get anywhere with this.

I think if you wanted to try and implement this, we could switch all the calls from __call to Finder.find and just explicitly call the function, rather than calling a metamethod.

Sorry I never got around to doing this (or replying).

[ltratt]

In the interim, I must admit that I ended up switching to LuaJIT, because I didn't want to give up on Telescope!

[tjdevries]

Glad to hear it's working for you at least!

[Nis5l]

Do you have ripgrep installed? @Nis5l

Yes i do, checkhealth recognized it as well:

# rg --version
ripgrep 13.0.0
-SIMD -AVX (compiled)
+SIMD +AVX (runtime)

For now i use commit ec5b6abbe0f06391098000af8d4829e941774b7c which works for me. I'm not exactly sure which commit breaks it for me, but i used this one because it was mentioned in #1292

[mrusme]

I can confirm that rebuilding nvim to use LuaJIT fixes this issue. On Gentoo, add the following package.use (assuming there was lua5-1 used as target):

app-editors/neovim lua_single_target_luajit -lua_single_target_lua5-1
dev-lua/luv lua_single_target_luajit -lua_single_target_lua5-1

[Nis5l]

Works as expected now, thank you very much :+1:

[tex]

Hello, any update to this issue? Unfortunately I am not familiar with LUA too much.

[tex]

And yes, using luajit instead of lua fixes the problem.

[eug-vs]

Same for me, using luajit fixes the issue 👍🏼

[lcheylus]

Same for me on Debian Linux testing amd 64 : rebuild of Neovim 0.7.0 with LuaJIT fixes this issue => Telescope works correctly with find_files and live_grep commands.

[naveepr]

How do we rebuild neovim using Luajit? Is it similar to as mentioned in https://github.com/neovim/neovim/wiki/Installing-Neovim#install-from-source. I am facing similar issue with neovim plugins failing with lua error.