Allow use within busted unit tests

[dainkaplan]

Not sure on the technical hurdles here, but being able to run a debugger within your unit tests would be awesome.

Busted allows calling the runner from within a regular lua file, so the launcher should be possible:

require 'busted.runner'()

describe("a test", function()
  -- tests to here
end)

(see documentation here: https://olivinelabs.com/busted/#usage)

And running it:

lua test.lua

However, when trying to combine the debugger and the above, the debugger gets halted before the first test is actually run. Could be due a limitation of lua51, not sure.

[tomblind]

I've attempted to reproduce this, but it seems to work fine for me. 😄

What happens when the debugger 'halts'? Is there an error? Can you set "verbose": true in the launch.json and post the log from the debug console?

[dainkaplan]

I’ll check the latest and see what whats what! Will report back. Thanks for looking into it, @tomblind

[astrochili]

@tomblind I can provide the case. My lunch.json:

{
    "configurations": [     
        {
            "name": "Test",
            "type": "lua-local",
            "request": "launch",
            "program": {
                "command": "busted"
            },
            "args": [
                "${workspaceFolder}/test/start.lua"
            ],
            "verbose": true
        }
    ]
}

My test/start.lua:

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

describe("a test", function()
  -- tests to here
end)

Busted falls on describe() or it() call. This is verbose output:

[request] launchRequest
[request] setBreakPointsRequest
[request] configurationDoneRequest
[info] launching `"busted" /Users/.../test/start.lua` from "/Users/..."
[info] process launched
[request] threadsRequest
[message] {"tag":"$luaDebug","message":"step","type":"debugBreak","threadId":1,"breakType":"step"}
[command] break clear
[message] {"tag":"$luaDebug","type":"breakpoints","breakpoints":[]}
[command] break set /usr/local/share/lua/5.3/busted/init.lua:65
[message] {"tag":"$luaDebug","type":"breakpoints","breakpoints":[{"file":"/usr/local/share/lua/5.3/busted/init.lua","enabled":true,"line":65}]}
[command] cont
[message] {"tag":"$luaDebug","message":"\nstack traceback:\n\t/usr/local/share/lua/5.3/busted/core.lua:268: in global 'describe'\n\t/Users/.../test/start.lua:5: in main chunk\n\t[C]: in function 'xpcall'\n\t/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'\n\t/usr/local/share/lua/5.3/busted/block.lua:146: in field 'execute'\n\t/usr/local/share/lua/5.3/busted/init.lua:7: in upvalue 'executor'\n\t/usr/local/share/lua/5.3/busted/core.lua:312: in function </usr/local/share/lua/5.3/busted/core.lua:312>\n\t[C]: in function 'xpcall'\n\t/usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'\n\t/usr/local/share/lua/5.3/busted/core.lua:312: in field 'execute'\n\t/usr/local/share/lua/5.3/busted/execute.lua:58: in local 'execute'\n\t/usr/local/share/lua/5.3/busted/runner.lua:197: in function 'busted.runner'\n\t/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted:3: in main chunk\n\t[C]: in ?","type":"debugBreak","threadId":1,"breakType":"error"}
[error] 
stack traceback:
    /usr/local/share/lua/5.3/busted/core.lua:268: in global 'describe'
    /Users/.../test/start.lua:5: in main chunk
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'
    /usr/local/share/lua/5.3/busted/block.lua:146: in field 'execute'
    /usr/local/share/lua/5.3/busted/init.lua:7: in upvalue 'executor'
    /usr/local/share/lua/5.3/busted/core.lua:312: in function </usr/local/share/lua/5.3/busted/core.lua:312>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.3/busted/core.lua:178: in field 'safe'
    /usr/local/share/lua/5.3/busted/core.lua:312: in field 'execute'
    /usr/local/share/lua/5.3/busted/execute.lua:58: in local 'execute'
    /usr/local/share/lua/5.3/busted/runner.lua:197: in function 'busted.runner'
    /usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted:3: in main chunk
    [C]: in ?
[request] threadsRequest
[command] threads
[message] {"tag":"$luaDebug","threads":[{"active":true,"id":1,"name":"main thread"}],"type":"threads"}
[request] stackTraceRequest 0/20 (thread 1)
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[request] evaluateRequest items
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] frame 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/./test/start.lua","line":4..4},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] eval items
[message] {"tag":"$luaDebug","type":"result","result":{"type":"nil","value":"nil"}}
[request] scopesRequest
[command] thread 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[command] frame 1
[message] {"tag":"$luaDebug","frames":[{"line":268,"active":true,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"describe"},{"source":"/Users/.../test/start.lua","line":5},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":146,"source":"/usr/local/share/lua/5.3/busted/block.lua","func":"execute"},{"line":7,"source":"/usr/local/share/lua/5.3/busted/init.lua","func":"executor"},{"source":"/usr/local/share/lua/5.3/busted/core.lua","line":312},{"line":-1,"source":"[C]","func":"xpcall"},{"line":178,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"safe"},{"line":312,"source":"/usr/local/share/lua/5.3/busted/core.lua","func":"execute"},{"line":58,"source":"/usr/local/share/lua/5.3/busted/execute.lua","func":"execute"},{"source":"/usr/local/share/lua/5.3/busted/runner.lua","line":197},{"source":"/usr/local/lib/luarocks/rocks-5.3/busted/2.0.0-1/bin/busted","line":3},{"source":"[C]","line":-1}],"type":"stack"}
[request] variablesRequest locals
[command] locals
[message] {"tag":"$luaDebug","variables":[{"type":"nil","value":"nil","name":"trace"},{"type":"table","length":0,"value":"[table: 0x7f8612651570]","name":"ctx"},{"type":"function","value":"[function: 0x7f861245b1e0]","name":"fn"},{"type":"string","value":"\"Test case\"","name":"name"}],"type":"variables"}

By the way, without this code

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

busted runs fine from VSCode, but without debugger of course.

[tomblind]

Ok, I do get the error using the busted cli. It turns out it's not an error at all: busted is using lua's debug library to get a callstack and this falsely triggers a debug break, because that's one way the debugger detects errors when running code outside of an xpcall. If you hit F5 and continue, the test script will still run and can be debugged.

The easiest workaround for this right now is to run busted using a lua interpreter instead of the busted cli, since this uses xpcall for error trapping. So, in your, launch.json:

"program": {
    "lua": "lua",
    "file": "${workspaceFolder}/test/start.lua"
}

and then in start.lua:

require("busted.runner")()

It looks like this is how @dainkaplan was doing it originally though, so I'm not sure why the error occurred for him.

I'll have to think on this more, since it would be good to support running from the custom cli.

[dainkaplan]

I haven’t been able to retest yet but hope to get to this finally this weekend! As @tomblind said that’s exactly how I was trying to run it. You’re using lua51?

[tomblind]

Yup - lua51

Make sure you don't use

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

when running this way. It's not necessary when using the lua interpreter and could cause this issue to still arise.

[astrochili]

@tomblind @dainkaplan Guys, I tried your suggestion to use lua lunching instead CLI. This is what happens: https://drive.google.com/file/d/1H_tJAbdvA9iXIyEoWWY_fQL6JxMCyWIl/view?usp=sharing

I use Lua 5.3 and can't test it with Lua 5.1 at this moment. Any ideas?

[tomblind]

I can't access the doc - can you post it here?

[astrochili]

I can't access the doc - can you post it here?

That's a video, I updated the sharing settings, try again please :)

[tomblind]

Oops. The environment isn't being set properly. I have a fix for that but haven't pushed it yet. Sorry 😄

[tomblind]

Actually I was wrong - the environment issue should not a problem when running in lua-interpreter mode.

Looking more closely at your video - there's something strange going on in the call stack. It looks like the debugger is rehooking a second time, which is probably messing up the environment. I have no idea off the top of my head why that would happen, though. It could be a lua5.3 thing - I'll have to figure out how to get that working and see.

[astrochili]

Actually I was wrong - the environment issue should not a problem when running in lua-interpreter mode.

Looking more closely at your video - there's something strange going on in the call stack. It looks like the debugger is rehooking a second time, which is probably messing up the environment. I have no idea off the top of my head why that would happen, though. It could be a lua5.3 thing - I'll have to figure out how to get that working and see.

I'd like to help you with testing. I tried to install busted via luarocks for lua 5.1 but without success.

[tomblind]

I managed to get busted working with 5.3 and reproduce your issue. Unfortunately, I don't think debugging using the interpreter in this way will work. When busted.runner is required, it re-evaluates any code passed to lua's command line, which runs a second debugger hook. Unfortunately, this second hook doesn't get access to the busted environment, hence the error.

However, I have fixed the original issue that occurred when running via the busted cli (6ac5b8663ee3953c1b898f8d5348056723244713) and it will be in the next release (which I hope to do today or tomorrow). With this fix, you can also run using a lua interpreter, but you have to set it up as a custom environment (use "command": "lua" in launch.json instead of "lua": "lua").

[astrochili]

@tomblind sounds good. I’ll be back with feedback when the update will be available.

[astrochili]

@tomblind I’d like to say again that your extension is the only one that works from the all that I tried. I don't know what kind of magic it is, but thank you! I actively use it.

[tomblind]

Thanks! My goal is make it as easy to use for common cases as possible, since most of the other debuggers available require dependencies and setup just to get them to work at all.

I've just pushed the new release. Let me know if it works for you.

[dainkaplan]

@tomblind With your latest changes I've confirmed I can run a test with:

      "program": {
        "command": "busted"
      },
      "args": ["spec/index_spec.lua"]

However, only breakpoints in the file passed as an argument to busted seems to work. Within the index_spec.lua, I have some require("other_test") to run the other tests, and all code included via require() seems to be entirely ignored.

Not sure on the source of this error yet, though! Will investigate a bit more.

[tomblind]

How does that work? When I try to require additional tests from the first, I get attempt to call a nil value (global 'describe') in the second script (not even with the debugger - just running busted from the command line).

[astrochili]

@tomblind @dainkaplan I confirm. This is my configuration that works fine. Breakpoints work, busted works. Thanks!

launch.json:

{
    "configurations": [
        {
            "name": "Busted",
            "type": "lua-local",
            "request": "launch",
            "program": {
                "command": "busted"
            },
            "args": [
                "${workspaceFolder}/test/start.lua"
            ],
        }
    ]
}

start.lua:

if os.getenv("LOCAL_LUA_DEBUGGER_VSCODE") == "1" then
    require("lldebugger").start()
end

describe("a test", function()
  -- tests to here
end)

[tomblind]

@dainkaplan Did you ever find out what was going on with the additional required tests? I was still never able to get that to work with busted in general. If that's something busted supports, I'd like to get it working with the debugger as well.

[mbsteixeira]

I also do not get to debug any lua lib.

If I execute the code below :

local debug = require("lldebugger") debug.start() --- debug ok debug.stop() -- try to debug here I also do not get to debug the debug.stop() execution .

[tomblind]

@mbsteixeira I'm not sure I understand. Once debug.stop() is called, you cannot debug anymore. That is expected behavior. In any case, it would be best to open a new issue for this.