[QUESTION]: How to avoid `attempt to yield across C-call boundary`

[pysan3]

NeoVim Version

NVIM v0.10.0-dev-2632+ge3bd04f2a-dirty
Build type: RelWithDebInfo
LuaJIT 2.1.1710088188
Run "nvim -V1 -v" for more info

Describe the bug

When table.sort is called inside a nio task (async context) and a nio async function is called in the compare function, I get attempt to yield across C-call boundary error.

Is there an easy way you can think of to avoid this problem? I'm very aware this is a VERY niche usecase but I wanted to at least bring this up to the discussion.

I can easily use vim.loop.fs_stat instead of nio.uv.fs_stat inside the compare function. TBH I think vim.loop solution is faster in this case since it does not need to send data back and forth.

I don't consider this a bug but hopefully comes up in the search engine if someone else has also been stuck with the same problem. If you can think of other workaround ideas, I'm also happy to hear them.

To Reproduce

local nio = require("nio")

nio.run(function()
  local paths = {}
  local cwd = vim.fn.getcwd()
  -- get all files in cwd
  local h_err, handler = nio.uv.fs_scandir(cwd)
  vim.print(string.format([[h_err: %s]], vim.inspect(h_err)))
  vim.print(string.format([[handler: %s]], vim.inspect(handler)))
  assert(not h_err and handler, string.format([[Invalid handler for %s: %s]], cwd, h_err))
  while true do
    local name, fs_type = vim.loop.fs_scandir_next(handler)
    if not name or not fs_type then
      break
    end
    table.insert(paths, cwd .. "/" .. name)
  end
  -- print paths
  for _, path in ipairs(paths) do
    vim.print(string.format([[path: %s]], path))
  end
  vim.print(string.format([[#paths: %s]], #paths))
  -- sort by create time
  table.sort(paths, function(a, b)
    local a_err, a_stat = nio.uv.fs_stat(tostring(a))
    local b_err, b_stat = nio.uv.fs_stat(tostring(b))
    assert(not a_err and a_stat, string.format([[Invalid stat: %s: %s]], a, a_err))
    assert(not b_err and b_stat, string.format([[Invalid stat: %s: %s]], b, b_err))
    return a_stat.ctime.sec < b_stat.ctime.sec
  end)

  -- print paths
  for _, path in ipairs(paths) do
    vim.print(string.format([[path: %s]], path))
  end
end)

Logs

  Error executing luv callback:
  .../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:95: Async task failed without callback: The coroutine failed with this message:
  .../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:198: attempt to yield across C-call boundary
  stack traceback:
    [C]: in function 'sort'
    [string ":source (no file)"]:84: in function <[string ":source (no file)"]:63>
  stack traceback:
    [C]: in function 'error'
    .../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:95: in function 'close_task'
    .../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:117: in function 'cb'
    .../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:183: in function <.../.local/share/nvim/lazy/nvim-nio/lua/nio/tasks.lua:182>

[rcarriga]

Yeah this makes sense as the sorting function is called from C, which means you can't yield from a coroutine. I think the best way to work around this would be to pre-compute the sortkeys (i.e. nio.uv.fs_stat(tostring(value))) and just look them up in the sort function. This also has the added benefit of not running the stat function multiple times for the same path and also not blocking the main thread as you would by using vim.loop

local nio = require("nio")

nio.run(function()
  local paths = {}
  local cwd = vim.fn.getcwd()
  -- get all files in cwd
  local h_err, handler = nio.uv.fs_scandir(cwd)
  vim.print(string.format([[h_err: %s]], vim.inspect(h_err)))
  vim.print(string.format([[handler: %s]], vim.inspect(handler)))
  assert(not h_err and handler, string.format([[Invalid handler for %s: %s]], cwd, h_err))
  while true do
    local name, fs_type = vim.loop.fs_scandir_next(handler)
    if not name or not fs_type then
      break
    end
    table.insert(paths, cwd .. "/" .. name)
  end
  -- print paths
  for _, path in ipairs(paths) do
    vim.print(string.format([[path: %s]], path))
  end
  vim.print(string.format([[#paths: %s]], #paths))

  local stats = {}
  for _, path in ipairs(paths) do
    local err, result = nio.uv.fs_stat(tostring(path))
    assert(not err, string.format([[Invalid stat: %s: %s]], path, err))
    stats[path] = result
  end
  -- sort by create time
  table.sort(paths, function(a, b)
    return stats[a].ctime.sec < stats[b].ctime.sec
  end)

  -- print paths
  for _, path in ipairs(paths) do
    vim.print(string.format([[path: %s]], path))
  end
end)

[pysan3]

Yah, I think that's the best solution. Thanks for your insight! 😆