PandocCouldNotFindDataFileError "lunajson.lua"

[njbart]

A change to the latest dev version of pandoc (possibly connected with this commit) seems to break the pandoc-zotxt.lua filter.

MWE (prerequisites: Zotero is running, and @author:2020title is a valid citekey):

$ echo @author:2020title | /usr/local/bin/pandoc -L pandoc-zotxt.lua -F pandoc-citeproc

using a pandoc binary installed via brew into /usr/local/bin/ generates the expected (HTML-)formatted in-text citation plus bibliography entry.

The latest dev version of pandoc, however, invoked with

$ echo @author:2020title | pandoc -L pandoc-zotxt.lua -F pandoc-citeproc

results in

Error running filter /Users/nick/.local/share/pandoc/filters/pandoc-zotxt.lua:
PandocCouldNotFindDataFileError "lunajson.lua"
stack traceback:
    [C]: in ?
    [C]: in function 'require'
    /Users/nick/.local/share/pandoc/filters/pandoc-zotxt.lua:179: in main chunk

(pandoc-citeproc is the latest dev version in both cases.)

Any ideas?

[odkr]

No good ones. Which version do you mean by "latest dev version"? Pandoc's master branch as per May 4? I just tried to compile 8fc5766 (most recent), d3ec53b (May 4), and 97fe2ea16 (May 3), but cabal install fails to install haddock in every case. How did you get it to compile?

[njbart]

I’ve been trying 8fc5766 and the currently most recent 82eb4df28 (resulting in the same error as described above). As to installing, I’m following the instructions at https://github.com/jgm/pandoc/wiki/Installing-the-development-version-of-pandoc, using stack rather than cabal-install.

[odkr]

Thanks for the pointer. I can re-predouce the error. pandoc-citeproc is not involved at all, which was to expect:

Calling

echo @adorno1967SociologyPsychologyPart | ./pandoc -L pandoc-zotxt.lua -F pandoc-citeproc

outputs:

Error running filter /Users/odin/.pandoc/filters/pandoc-zotxt.lua: PandocCouldNotFindDataFileError "lunajson.lua" stack traceback: [C]: in ? [C]: in function 'require' /Users/odin/.pandoc/filters/pandoc-zotxt.lua:179: in main chunk bin$ echo @adorno1967SociologyPsychologyPart | ./pandoc -L pandoc-zotxt.lua Error running filter /Users/odin/.pandoc/filters/pandoc-zotxt.lua: PandocCouldNotFindDataFileError "lunajson.lua" stack traceback: [C]: in ? [C]: in function 'require' /Users/odin/.pandoc/filters/pandoc-zotxt.lua:179: in main chunk

I've created a debug version of the filter, called pandoc-zotxt-debug.lua, with the additional line

io.stderr:write(PANDOC_SCRIPT_FILE, '\n', package.path, '\n')

above the local json = require 'lunajson' statement.

Calling

echo @adorno1967SociologyPsychologyPart | ./pandoc -L pandoc-zotxt-debug.lua

outputs:

/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua /usr/local/Cellar/luarocks/3.2.1/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;./?.lua;./?/init.lua;/Users/odin/.luarocks/share/lua/5.3/?.lua;/Users/odin/.luarocks/share/lua/5.3/?/init.lua;/Users/odin/.pandoc/filters/share/lua/5.3/?.lua;/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua-0.3.17/share/lua/5.3/?.lua Error running filter /Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua: PandocCouldNotFindDataFileError "lunajson.lua" stack traceback: [C]: in ? [C]: in function 'require' /Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua:181: in main chunk

That is, the script locates itself (/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua) and adds the correct patterns ( /Users/odin/.pandoc/filters/share/lua/5.3/?.lua;/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua-0.3.17/share/lua/5.3/?.lua) to package.path.

By contrast, calling

echo @adorno1967SociologyPsychologyPart | pandoc -L pandoc-zotxt-debug.lua
/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua

outputs:

/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua /usr/local/Cellar/luarocks/3.2.1/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;./?.lua;./?/init.lua;/Users/odin/.luarocks/share/lua/5.3/?.lua;/Users/odin/.luarocks/share/lua/5.3/?/init.lua;/Users/odin/.pandoc/filters/share/lua/5.3/?.lua;/Users/odin/.pandoc/filters/pandoc-zotxt-debug.lua-0.3.17/share/lua/5.3/?.lua \<p>\<span class="citation" data-cites="adorno1967SociologyPsychologyPart">@adorno1967SociologyPsychologyPart\</span>\</p>

For all I can tell, this is a bug in the development version of Pandoc. I'll file an issue with them.

[odkr]

Drilled it down to the essentials:

Two files: require-debug.lua and test.lua. These need to be placed in the same directory.

require-debug.lua:

local concat = table.concat
local unpack = table.unpack
local stderr = io.stderr

PATH_SEP = package.config:sub(1, 1)

do
    local split = '(.-' .. PATH_SEP .. '?)([^' .. PATH_SEP .. ']-)$'
    local sanitisers = {{PATH_SEP .. '%.' .. PATH_SEP, PATH_SEP},
        {PATH_SEP .. '+', PATH_SEP}, {'^%.' .. PATH_SEP, ''}}

    function split_path (path)
        assert(path ~= '', 'path is the empty string')
        for _, s in ipairs(sanitisers) do path = path:gsub(unpack(s)) end
        local dir, fname = path:match(split)
        dir = dir:gsub('(.)' .. PATH_SEP .. '$', '%1')
        if dir == '' then dir = '.' end
        if fname == '' then fname = '.' end
        return dir, fname
    end
end

local wd = split_path(PANDOC_SCRIPT_FILE)
stderr:write('working directory: ', wd, '\n')
package.path = concat({package.path, concat({wd, '?.lua'}, PATH_SEP)}, ';')
stderr:write('package.path: ', package.path, '\n')

local test = require 'test'

test.lua is empty; it just has to exist.

Then:

pandoc -L require-debug.lua </dev/null

outputs:

pandoc -L require-debug.lua </dev/null working directory: /Users/odin/.pandoc/filters package.path: /usr/local/Cellar/luarocks/3.2.1/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;./?.lua;./?/init.lua;/Users/odin/.luarocks/share/lua/5.3/?.lua;/Users/odin/.luarocks/share/lua/5.3/?/init.lua;/Users/odin/.pandoc/filters/?.lua

By contrast, calling

./pandoc -L require-debug.lua </dev/null

outputs:

working directory: /Users/odin/.pandoc/filters package.path: /usr/local/Cellar/luarocks/3.2.1/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?.lua;/usr/local/share/lua/5.3/?/init.lua;/usr/local/lib/lua/5.3/?.lua;/usr/local/lib/lua/5.3/?/init.lua;./?.lua;./?/init.lua;/Users/odin/.luarocks/share/lua/5.3/?.lua;/Users/odin/.luarocks/share/lua/5.3/?/init.lua;/Users/odin/.pandoc/filters/?.lua Error running filter /Users/odin/.pandoc/filters/require-debug.lua: PandocCouldNotFindDataFileError "test.lua" stack traceback: [C]: in ? [C]: in function 'require' /Users/odin/.pandoc/filters/require-debug.lua:28: in main chunk

pandoc refers to Pandoc version 2.9.2.1. ./pandoc refers to the development version as per commit 8fc5766.

Where pandoc refers to Pandoc version 2.9.2.1.

[odkr]

Why do you suspect commit 0fafe9dd32d5a0266d74ac78b1818050bc69a4bd? (But maybe answer over at https://github.com/jgm/pandoc/issues/6361)

[odkr]

Filed at https://github.com/jgm/pandoc/issues/6361.

[odkr]

It is a bug in the development version of Pandoc and will be fixed.

[njbart]

Works nicely again now, many thanks for having looked into this.