wishlist: Lua REPL and other dev/debug utilities

[inklesspen]

Since the Pandoc module is created via Haskell code, it is impossible (as far as I am aware) to run the Lua REPL and work with Pandoc objects. This means that my development/debugging attempts have to mostly use Penlight utilities to log tables to stdout/stderr.

It would make filter development a lot easier if there was a way to break into a Lua REPL at any particular point in the filter.

A utility function to log the structure of a Pandoc object would also be very handy.

[iandol]

This would be wonderful. For many of us who are not programmers, Lua is understandable enough but the data structures impenetrable enough that a REPL would really really help. I had previously tried to get remote debugging working with Zerobrane Studio last year, perhaps that now works:

https://groups.google.com/g/pandoc-discuss/c/mqUq4-aqBnA/m/txZBaaFLBAAJ

@inklesspen — I know little about Lua, what are penlight utilities and how can they be used in Lua filters?

[alerque]

Penlight is just a 3rd party Lua library suppling various useful utility functions. It's common for many projects and programmers to include it since Lua itself takes a no-batteries-included approach. Another common one is stdlib (which in spite of the name is not actually Lua's stdlib and is now pretty much unmaintained). Penlight is available on LuaRocks or through many distro's package managers. Among the utilities it provides are functions useful for debugging such as table dump and pretty-formattters. If you install it to your system or add it to your project you can import it and use these utilities in your Lua filters.

[iandol]

Thanks!

So does anyone with some Lua-fu have experience with remote debugging — if it can be made to work with Lua filters in Pandoc then this would solve the REPL/Debug issue (remote debugging works great for Pandoc filters written in Ruby).

Because of its nature as an embedded language, there is rich remote debugging support for Lua: https://studio.zerobrane.com/doc-remote-debugging — so I suspect my inability to get it to work last year is due my own ignorance of Lua.

[alerque]

I did try once briefly, but don't use Zero Brane Studio more than sporadically and didn't try very hard. I think we could page @pkulchenko on this one though. He would probably know if there was a way to wire it up as a remote debugger for a Pandoc filter. Or maybe it would be more on-topic to save this issue for actual Pandoc REPL commentary and open a new issue on zbstudio's issue tracker about how to configure it for this.

[pkulchenko]

@iandol, I suspect it might be possible, especially given that you got some things working. It seems like the issues you had were mostly around breakpoints and stepping not working in sub-functions. We can try to troubleshoot it here or open an issue in ZeroBrane studio tracker as @alerque suggested (Thanks Caleb!).

There are couple of things that come to mind: the files that have functions you are stepping into need to be open in the IDE or you need to have editor.autoactivate=true set in the config (see the documentation for it). The breakpoints not firing usually point to a mapping issue between file paths in the debugger and the IDE (see the third item on the list in the FAQ for details on how to troubleshoot).

There have been several improvements for directory mapping recently, so you may want to check the current master branch. I'd need some details about the project/path structure and content of the Output window when you start debugging.

[iandol]

I just tested (macOS, homebrew for installing pandoc/lua/zerobrane) and it now works out-of-the-box! 😀 It works without needing to modify LUA_PATH and LUA_CPATH (I did install mobdebug, socket, penlight etc. into my separately installed Lua 5.3), and running a test filter:

test.lua

pp=require('pl.pretty')
function Emph(elem)
  require("mobdebug").start()
  pp.dump(elem)
  return elem.content
end
function Strong(elem)
  require("mobdebug").start()
  pp.dump(elem)
  return pandoc.SmallCaps(elem.content)
end

> pandoc --lua-filter test.lua
Here is a *test* for the **filter**.
[CTRL+D]

And the debugger stops in the Emph function and I can inspect and modify the state and variables, run commands in the context of the script etc. The only problem is that the require("mobdebug").start() in the second function never triggers (I naively expect if I "continue" when I am in Emph() then it should stop me in Strong()), but as long as you step you can get into that function too.

So a REPL with lots of debugging tools is viable for developing Lua filters in Pandoc 👍

Thanks @pkulchenko for a really nice and flexible tool.

[pkulchenko]

I just tested (macOS, homebrew for installing pandoc/lua/zerobrane) and it now works out-of-the-box! grinning

That's great to hear!

The only problem is that the require("mobdebug").start() in the second function never triggers (I naively expect if I "continue" when I am in Emph() then it should stop me in Strong()), but as long as you step you can get into that function too.

Yes, start() will only work the first time when you initiate the debugging session (and won't do anything when called the second time). You can use require("mobdebug").pause() instead to stop debugging where you need (it acts similar to a breakpoint set on that line).

[jgm]

This is good to hear. Perhaps we should include a writeup about debugging techniques in the docs for lua filters?

[iandol]

I'm happy to help where I can, though my relative ignorance of Lua means much of this is indistinguishable from magic to me ;-) In addition we could add a luatest.lua filter to https://github.com/pandoc/lua-filters with the description for setup in the readme and commenting in code itself? Here are my self-instructions: https://github.com/iandol/dotpandoc/blob/master/filters/luatest.lua

[jgm]

@tarleb what is the main obstacle now to supporting an interactive mode (repl) with pandoc lua?

[tarleb]

If I remember correctly, then the only issue is that I hit a minor issue and then got sidetracked while writing a haskeline-based REPL for hslua-cli. Here's the branch: https://github.com/hslua/hslua/tree/hslua-cli-repl

[tarleb]

I took another look today, but ran into judah/haskeline#69.

[jgm]

What about isocline? It claims to have a multiline mode. Seems to be small and portable (and small dependency footprint).

[tarleb]

I'll give it a try :+1:

[tarleb]

Some progress: #8700.

[tarleb]

About starting a REPL in arbitrary places: it is unfortunately non-trivial to pass the local environment to the REPL. I.e., if we have

function Pandoc (doc)
  return pandoc.cli.repl()
end

then the name doc is not bound in the repl. My current approach is to allow passing a table that's automatically amended with the global environment, so one could write return pandoc.cli.repl { doc = doc }, but I'm not sure that's satisfactory.

[jgm]

return pandoc.cli.repl { doc = doc }

I think this would be sufficient, personally!

Unrelated question: while fooling with the repl, I sometimes wished I could write Str "hi" instead of pandoc.Str "hi" and so on. That is, I wanted to be able to do something like import pandoc (Str, Emph, Strong, read, write). Have you ever thought about providing something like this? Or is it already possible?

EDIT: I guess the Lua for this is just

local Str, Emph, Strong, read, write  = pandoc.Str, pandoc.Emph, pandoc.Strong, pandoc.read, pandoc.write

But it still seems that people who are developing pandoc filters might want to have a command that brings all the things in the pandoc namespace into the top-level namespace. Bad idea?

[tarleb]

OK, I've opened #8703 for this.

My personal, slightly low-brow, approach for easy access in the REPL would be to place a file panglob.lua with the following content somewhere in the LUA_PATH:

for k, v in pairs(pandoc) do
  _G[k] = v
end

Then I'd call pandoc with

pandoc lua -l panglob -i

(Needs the commit that I just pushed; there was a bug in hslua-cli that prevented this from working.)

[tarleb]

I sometimes do something like this in very long filters

local ipairs, load, pairs, type = ipairs, load, pairs, type
local string, table = string, table
_ENV = pandoc

But that doesn't work in the repl.

[tarleb]

I started work on a describe function that should print helpful info about an object. The docs for pandoc functions are available in the Lua interpreter, which would allow us to print a detailed description for describe(pandoc.read) and similar requests.