tests/busted: `LuaJIT ASSERT lj_api.c:836: lua_rawgeti: stack slot 2 is not a table`

andmis commented 2 years ago

Neovim version (nvim -v)

Latest as of right now, cb3211516298c8b01ffb432b7a7f33acdc01b34b

Vim (not Nvim) behaves the same?

N/A

Operating system/version

macOS 12

Terminal name/version

iTerm2

$TERM environment variable

xterm-256color

Installation

From source

How to reproduce the issue

Clean clone of the repo, run TEST_FILTER=whatever make test.

Expected behavior

Clean run of the tests.

Actual behavior

[----------] Global test environment teardown.
[==========] 12 tests from 365 test files ran. (2239.55 ms total)
[  PASSED  ] 12 tests.
-- Tests exited non-zero: Subprocess aborted
-- Output to stderr:
LuaJIT ASSERT lj_api.c:836: lua_rawgeti: stack slot 2 is not a table

CMake Error at /Users/mishchea/tmp/neovim/cmake/RunTests.cmake:98 (message):
  functional tests failed with error: Subprocess aborted

FAILED: CMakeFiles/functionaltest /Users/mishchea/tmp/neovim/build/CMakeFiles/functionaltest
cd /Users/mishchea/tmp/neovim/build && /opt/homebrew/Cellar/cmake/3.23.1/bin/cmake -DBUSTED_PRG=/Users/mishchea/tmp/neovim/.deps/usr/bin/busted -DLUA_PRG=/Users/mishchea/tmp/neovim/.deps/usr/bin/luajit -DNVIM_PRG=/Users/mishchea/tmp/neovim/build/bin/nvim -DWORKING_DIR=/Users/mishchea/tmp/neovim -DBUSTED_OUTPUT_TYPE=nvim -DTEST_DIR=/Users/mishchea/tmp/neovim/test -DBUILD_DIR=/Users/mishchea/tmp/neovim/build -DTEST_TYPE=functional -P /Users/mishchea/tmp/neovim/cmake/RunTests.cmake
ninja: build stopped: subcommand failed.
make: *** [functionaltest] Error 1
~/repos/neovim $

clason commented 2 years ago

Ah, the winhighlight is a red herring; the crash is in the test harness itself (it also happens with, e.g., TEST_FILTER=autocmd or even TEST_FILTER=sjflksflsj). It doesn't happen with TEST_FILE, though.

So this is in fact an issue with busted (or LuaJIT -- where such crashes tend to be fixed very quickly once a minimal standalone reproducer can be provided).

andmis commented 2 years ago

Yep, I'm on an Apple Silicon M1, and yeah this is independent of the value of TEST_FILTER=.

Also I should say, I'm pretty sure this is a recent regression. I used to be able to run the test suite on this machine.

clason commented 2 years ago

Also I should say, I'm pretty sure this is a recent regression. I used to be able to run the test suite on this machine.

If you can bisect it, that would be helpful. (I'd focus on the LuaJIT and LibUV bumps to get a starting point for the git bisect.)

andmis commented 2 years ago

Either I was mistaken that this ever worked for me, or something in my env changed that caused this to stop working. If I go back around 2300 commits from master (the oldest commit where git clean -f -x -d && make test works out-of-the-box for me, as in, doesn't give me a different error) I get the same error.

Some more data points:

TEST_FILTER=something make functionaltest works, but TEST_FILTER=doesnotexist make functionaltest does not -- the latter filter matches zero tests, the former matches some. So matching no tests seems to matter.
I just tried this in a Rosetta environment (x86 emulation for Apple Silicon) and got the same error.

clason commented 2 years ago

Yeah, I get the same on Intel.

So this looks increasingly like a busted issue, and not LuaJIT. If you can extract a minimal reproducing example using only busted (not Neovim), you could open an issue there. (It hasn't seen any real development in years, so I'm not optimistic about this getting fixed, mind you.)

neovim / neovim