Add Lua 5.1/5.2 test suites

[dibyendumajumdar]

This issue will track progress for adding the test suites from Lua (http://www.lua.org/tests/) to this repository. Modifications will be made to the test suites where necessary to make them work with LuaJIT.

It maybe possible to backport additional tests from 5.2.x and 5.3.x series where the language & libraries are compatible.

I will do the work on a branch initially and once working will merge it to the master.

License: The Lua Test suites are under MIT license.

[dibyendumajumdar]

Imported and hacked the Lua 5.1 and 5.2.2 test suites so that they run successfully on Mac OSX El Capitan. To be verified on Linux and Windows.

[lukego]

Is this ready to merge? (Should I make the CI in #10 run this test suite?)

[dibyendumajumdar]

I will do some more work (enabling more tests where possible) before merging. I also need to test on Windows. I have currently run them only against the master branch on Intel x64 platform so not sure if they will work or fail in other platforms. But yes it would be good to have this in CI once merged.

The 5.1 tests require different build options than 5.2 tests - and ideally both should have assertions and API checks enabled, are you able to do that in your CI configuration?

[lukego]

Roger. CI is flexible. Can also test all interesting permutations of build options for example.

[lukego]

(Just now it is only running tests on Linux/x64. There is probably a clever way to support other platforms via QEMU. Have no immediate plans to do that myself.)

[dibyendumajumdar]

@lukego hi, I have merged this now. Please let me know if you need me to do anything to make this work in CI. Thanks

[corsix]

and ideally both should have assertions and API checks enabled, are you able to do that in your CI configuration?

Ideally you'd test once with assertions and API checks enabled (to catch assertion failures) and once with them disabled (to test what is actually commonly used). As @lukego mentions, you'd also want to test on a bunch of architectures. You'd also want a few additional axes in the test matrix to cover various build-time options (5.2 compat, GC64, dual-num, no-jit, no-ffi). Last time I thought this through I came up with the following to enumerate the various combinations: (though it still doesn't cover operating system, and makes a few compromises to reduce the matrix size)

local arches = {
  x64 = {
    alt_num_mode = "dual",
    alt_gc64 = true,
  },
  x86 = {
    makeflags = [[CC="gcc -m32"]],
    alt_num_mode = "dual",
  },
  arm = {
    makeflags = [[HOST_CC="gcc -m32" CROSS=arm-linux-gnueabi- TARGET_CFLAGS="-mfloat-abi=soft"]],
    qemu = "qemu-arm",
  },
  armhf = {
    makeflags = [[HOST_CC="gcc -m32" CROSS=arm-linux-gnueabihf-]],
    qemu = "qemu-arm",
  },
  arm64 = {
    makeflags = [[CROSS=aarch64-linux-gnu-]],
    qemu = "qemu-aarch64",
  },
  mips = {
    xroot = "/home/cross/x-tools/mips-malta-linux-gnu",
    makeflags = [[HOST_CC="gcc -m32" CROSS=mips-malta-linux-gnu-]],
    qemu = "qemu-mips",
  },
  mipsel = {
    makeflags = [[HOST_CC="gcc -m32" CROSS=mipsel-linux-gnu-]],
    qemu = "qemu-mipsel",
  },
  ppc_e300 = {
    xroot = "/home/cross/x-tools/powerpc-e300c3-linux-gnu",
    makeflags = [[HOST_CC="gcc -m32" CROSS=powerpc-e300c3-linux-gnu-]],
    qemu = "qemu-ppc",
    alt_num_mode = "single",
  },
}

local function keys(t)
  local a = {}
  for k in pairs(t) do a[#a+1] = k end
  table.sort(a)
  return a
end

for _, k in ipairs(keys(arches)) do
  local ainfo = arches[k]
  arches[k .."_compat52"] = setmetatable({
    cflags = (ainfo.cflags or "") .." -DLUAJIT_ENABLE_LUA52COMPAT",
  }, {__index = ainfo})
end

for _, k in ipairs(keys(arches)) do
  local ainfo = arches[k]
  if ainfo.alt_gc64 then
    arches[k .."_gc64"] = setmetatable({
      cflags = (ainfo.cflags or "") .." -DLUAJIT_ENABLE_GC64",
      alt_gc64 = false,
    }, {__index = ainfo})
  end
end

for _, k in ipairs(keys(arches)) do
  local ainfo = arches[k]
  local alt = ainfo.alt_num_mode
  if alt then
    arches[k .."_".. alt:sub(1, 1) .."n"] = setmetatable({
      cflags = (ainfo.cflags or "") .." -DLUAJIT_NUMMODE=".. (alt == "single" and "1" or "2"),
      alt_num_mode = (alt == "single") and "dual" or "single",
    }, {__index = ainfo})
  end
end

for _, k in ipairs(keys(arches)) do
  local ainfo = arches[k]
  arches[k .."_nojitffi"] = setmetatable({
    cflags = (ainfo.cflags or "") .." -DLUAJIT_DISABLE_FFI -DLUAJIT_DISABLE_JIT",
  }, {__index = ainfo})
end

for _, k in ipairs(keys(arches)) do
  local ainfo = arches[k]
  arches[k .."_debug"] = setmetatable({
    cflags = (ainfo.cflags or "") .." -DLUAJIT_USE_GDBJIT -DLUA_USE_ASSERT",
    makeflags = (ainfo.makeflags or "") .." CCDEBUG=-g",
  }, {__index = ainfo})
end

hi, I have merged this now. Please let me know if you need me to do anything to make this work in CI. Thanks

Ugh, now we have three test runners, two of which aren't Windows friendly. I was really hoping you'd integrate the tests into the existing framework rather than just dump them in as totally separate things.

[dibyendumajumdar]

I think these tests are better off being a close mirror of standard Lua tests, rather than being modified to suit some framework.

[MikePall]

@dibyendumajumdar Nope, this is not helpful. Please split those tests up carefully, untangle the dependencies and integrate them properly. No debug spew, no big-ball-of-inseparable-tests and no separate test runners.

@corsix There are three ARM 32 bit test targets: soft-float ABI + soft-fp, soft-float ABI + VFP, hard-float ABI + VFP. Also, MIPS has both hard-float and soft-float now. And there's partial support for MIPS64.

[dibyendumajumdar]

@MikePall Apologies that is a big undertaking that I am unable to embark upon. I will delete the tests; I hope a simple delete followed by commit will be enough.