fs_scandir_next() sometimes returns name but no type

[troiganto]

Hi!

After debugging for a long time why the LuaSnip plugin for Neovim doesn't load all my snippets, I think I've traced the issue to this (otherwise excellent!) library.

The documentation for luv.fs_scandir_next() states that the function either returns two strings (file name and directory entry type) or nil or it fails completely.

However, for whatever reason, on the specific machine that I use, I receive the file name as usual but nil for the entry type. (MWE at the bottom.) Because downstream Lua scripts expect either two strings or two nils, this breaks a lot of their assumptions and code starts to behave strangely.

After some digging, I think I located the issue in fs.c:126: whenever ent->type is an unknown enum value, the type is set to the string "unknown" and two values are returned as usual. However, if ent->type is the known enum value UV_DIRENT_UNKNOWN, then the type is not set at all and only one value is returned.

I'm not sure what the correct behavior is here, but I think either the code should be changed (by merging the UV_DIRENT_UNKNOWN with the default) or the docs (to inform users that sometimes string, nil is returned).

Of course, if you can think of a simple reason why the directory entry type suddenly fails to be recognized, I'd be extremely grateful for any pointers. :slightly_smiling_face:

No matter how you decide, thanks for your time and hard work!

Minimal example (though I'm not sure how useful it is on most machines):

$ git clone https://github.com/luvit/luv.git --recursive
...
$ cd luv
$  git describe
1.40.0-0-148-gd15a473
$ make
...
$ cd build
$ ./luajit ~/mwe.lua

where mwe.lua:

function Main()
  local luv = require "luv"
  local root = "<HOME>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_scandir(root)
  local name, type = "", ""
  while name do
    name, type = luv.fs_scandir_next(fs)
    print(type, name)
  end
end

Main()

and the output is:

file    _.snippets
file    actionscript.snippets
file    ada.snippets
file    all.snippets
file    alpaca.snippets
file    apache.snippets
file    arduino.snippets
file    asm.snippets
file    autoit.snippets
file    awk.snippets
file    bash.snippets
file    c.snippets
file    chef.snippets
file    clojure.snippets
file    cmake.snippets
file    codeigniter.snippets
directory   coffee
file    cpp.snippets
nil crystal.snippets
nil cs.snippets
nil css.snippets
nil cuda.snippets
nil d.snippets
nil dart-flutter.snippets
nil dart.snippets
nil diff.snippets
nil django.snippets
nil dosini.snippets
nil eelixir.snippets
nil elixir.snippets
nil elm.snippets
nil erlang.snippets
nil eruby.snippets
nil falcon.snippets
nil fortran.snippets
nil freemarker.snippets
nil fsharp.snippets
nil gdscript.snippets
nil gitcommit.snippets
nil gleam.snippets
nil go.snippets
nil haml.snippets
nil handlebars.snippets
nil haskell.snippets
nil heex.snippets
nil helm.snippets
nil html.snippets
nil htmldjango.snippets
nil htmltornado.snippets
nil idris.snippets
nil jade.snippets
nil java.snippets
nil javascript
nil javascript-bemjson.snippets
nil javascript-d3.snippets
nil javascript-jasmine.snippets
nil javascript-mocha.snippets
nil javascript-openui5.snippets
nil jenkins.snippets
nil jinja.snippets
nil jsp.snippets
nil julia.snippets
nil kotlin.snippets
nil laravel.snippets
nil ledger.snippets
nil lfe.snippets
nil liquid.snippets
nil lpc.snippets
nil ls.snippets
nil lua.snippets
nil make.snippets
nil mako.snippets
nil markdown.snippets
nil matlab.snippets
nil mustache.snippets
nil objc.snippets
nil ocaml.snippets
nil octave.snippets
nil openfoam.snippets
nil org.snippets
nil pandoc.snippets
nil perl.snippets
nil perl6.snippets
nil phoenix.snippets
nil php.snippets
nil plsql.snippets
nil po.snippets
nil processing.snippets
nil progress.snippets
nil ps1.snippets
nil puppet.snippets
nil purescript.snippets
nil python.snippets
nil r.snippets
nil racket.snippets
nil rails.snippets
nil reason.snippets
nil rmd.snippets
nil rst.snippets
nil ruby.snippets
nil rust.snippets
nil sass.snippets
nil scala.snippets
nil scheme.snippets
nil scss.snippets
nil sh.snippets
nil simplemvcf.snippets
nil slim.snippets
nil smarty.snippets
nil snippets.snippets
nil sql.snippets
nil stylus.snippets
nil supercollider.snippets
nil svelte.snippets
nil systemverilog.snippets
nil tcl.snippets
nil tex.snippets
nil textile.snippets
nil twig.snippets
nil typescript.snippets
nil typescriptreact.snippets
nil verilog.snippets
nil vhdl.snippets
nil vim.snippets
nil vue.snippets
nil xml.snippets
nil xslt.snippets
nil yii-chtml.snippets
nil yii.snippets
nil zsh.snippets
nil nil

[SinisterRectus]

Odd that it works then stops working.

According to https://www.gnu.org/software/libc/manual/html_node/Directory-Entries.html:

The type is unknown. Only some filesystems have full support to return the type of the file, others might always return this value.

On what OS and filesystem do you see this happening?

[troiganto]

It's odd indeed, it took me a long time to piece together what was happening here.

I access the affected machine remotely, it's managed by our IT department. I'm in contact with them in parallel to figure out if there's something they can do.

$ cat /etc/redhat-release
Rocky Linux release 9.2 (Blue Onyx)
$ mount | grep "/home "
<...> on /home type nfs4 (rw,nosuid,nodev,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=<ip addr>,local_lock=none,addr=<ip addr>)

I wonder if the network filesystem and its inevitable lag cause any shenanigans somewhere inside libuv …

[squeek502]

Surprised that our docs don't mention this; that should definitely be fixed regardless of whether we return nil or "unknown" as the type.

I assume the same applies to fs_readdir but the Libuv docs don't mention it there. Will need to try to test that as well.

[Bilal2453]

We can make this return a "unknown" string as well, this is probably the more correct behavior here. I remember reading the code relating this, should be an easy fix, assuming it wasn't intended behavior this shouldn't be a breaking change as well.

[squeek502]

Personally, I think I prefer keeping it as nil, or at least keeping some way of distinguishing between 'we got a type but it's unknown--it is not one of the known types though' and 'we didn't get a type at all--it could be anything, you probably need to do a stat call to find out for sure'.

[troiganto]

I can confirm that fs_readdir() shows the same behavior:

function Main()
  local luv = require "luv"
  local root = "<home>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_opendir(root, nil, 200)
  local data = luv.fs_readdir(fs)
  if data then
    for _, dirent in ipairs(data) do
      print(dirent.type, dirent.name)
    end
  end
  if luv.fs_readdir(fs) then
    print("...")
  end
  luv.fs_closedir(fs)
end

Main()

This prints the same (lengthy) output as the original example.

Explicitly using stat on the entries that couldn't be read seems to kick the filesystem into doing its job. The following script correctly marks all files as "file" instead of nil:

function Main()
  local luv = require "luv"
  local root = "<home>/.local/share/nvim/lazy/vim-snippets/snippets"
  local fs = luv.fs_scandir(root)
  local name, type = "", ""
  while name do
    name, type = luv.fs_scandir_next(fs)
    if name and not type then
      local stat = luv.fs_stat(root .. "/" .. name)
      type = stat.type
    end
    print(type, name)
  end
end

Main()