Lua 5.3 doesn't have "incremental" option for garbage colector

[mingodad]

This commit https://github.com/edubart/nelua-lang/commit/ee033b0e8841ca686348ac7f883fdf5333c526bd#diff-bf189d5503a29964078ea21c21f4d9edacf0b8233037979ddfa501b7b790ab56 add the "incremental" parameter to the garbage collector but it's only valid for Lua 5.4

[edubart]

Nelua will start to give support for Lua 5.4+ only. I don't want to fragment the code, the issues, the users by giving support for multiple Lua versions. So not a bug.

[mingodad]

Thank you for your help ! Anyway I'm trying again with Lua-5.3.6 as an experiment and I'm puzzled by the few 4 tests that fail, when I run the failed tests through nelua in both this project nelua and in my variant nelua with Lua-5.3.6 plus your changes to Lua-5.4.4 plus utf8 I get the same output.

Some of my changes in lualib/nelua are to output the original sources when a test fail, I understand your point to only give support to Lua-5.4 but just in case this experiment can surface any latent bug and you could have a look at it I'll appreciate.

The reason I'm testing with Lua-5.3 is because I want to port nelua to use LJS https://github.com/mingodad/ljs that at the moment only works with Lua from 5.1 to 5.3 (5.4 is still moving a bit 5.4, 5.4.1, 5.4.2, 5.4.3, 5.4.4 and there is new bugs reported on the mailing list that are not yet on the probably comming 5.4.5).

My diff with the latest version of nelua:

diff --git a/lualib/nelua/runner.lua b/lualib/nelua/runner.lua
index d10f6c69..25df5ff6 100644
--- a/lualib/nelua/runner.lua
+++ b/lualib/nelua/runner.lua
@@ -6,12 +6,14 @@ it's the first required module when running the compiler.
 ]]

 -- We expect to be running in Lua 5.4.
-if _VERSION ~= 'Lua 5.4' then
-  error 'Please use Lua 5.4'
+if not (_VERSION == 'Lua 5.4' or _VERSION == 'Lua 5.3') then
+  error 'Please use Lua 5.4 or Lua 5.3'
 end

 -- Make the lua garbage collector less aggressive to speed up compilation.
-collectgarbage("incremental", 800, 400, 16)
+if _VERSION == 'Lua 5.4' then
+  collectgarbage("incremental", 800, 400, 16)
+end

 -- Timers must be the first loaded module.
 local nanotimer = require 'nelua.utils.nanotimer'
diff --git a/spec/tools/expect.lua b/spec/tools/expect.lua
index d091d275..0e801636 100644
--- a/spec/tools/expect.lua
+++ b/spec/tools/expect.lua
@@ -30,6 +30,16 @@ expect.config = { srcname = nil }

 function expect.same_string(expected, passedin)
   if expected ~= passedin then --luacov:disable
+print("===expected"); print(expected);print("===passedin"); print(passedin);print("===")
+    error('Expected strings to be the same, difference:\n' ..
+      differ(expected, passedin):tostring({colored = true, context=3}))
+  end --luacov:enable
+end
+
+function expect.same_string_code(expected, passedin, code, expected_code)
+  if expected ~= passedin then --luacov:disable
+print("===expected"); print(expected);print("===passedin"); print(passedin);print("===")
+print("===code"); print(code);print("===expected_code"); print(expected_code);print("===")
     error('Expected strings to be the same, difference:\n' ..
       differ(expected, passedin):tostring({colored = true, context=3}))
   end --luacov:enable
@@ -276,7 +286,7 @@ function expect.ast_type_equals(code, expected_code)
   local expected_ast = expect.analyze_ast(expected_code)
   filter_ast_for_check(ast)
   filter_ast_for_check(expected_ast)
-  expect.same_string(tostring(expected_ast), tostring(ast))
+  expect.same_string_code(tostring(expected_ast), tostring(ast), code, expected_code)
 end

 function expect.analyze_error(code, expected_error)
diff --git a/src/onelua.c b/src/onelua.c
index 3c605981..58cec98d 100644
--- a/src/onelua.c
+++ b/src/onelua.c
@@ -53,7 +53,12 @@
 #undef LUAI_DDEC
 #undef LUAI_DDEF
 #define LUAI_FUNC  static
+#ifdef LUAI_MAXALIGN /*Lua 5.4*/
 #define LUAI_DDEC(def) /* empty */
+#else /*Lua 5.3*/
+#undef LUA_USE_READLINE
+#define LUAI_DDEC  static
+#endif
 #define LUAI_DDEF  static

 /* core -- used by all */

Here is my Lua-5.3.6 plus your changes to Lua-5.4.4 lua-5.3.6-nelua.zip

And here is a shell script that test the failed tests with nelua:

#!/bin/sh
rm -rf $HOME/.cache/nelua

#nelua_args=--print-ast
nelua_args=--print-analyzed-ast

echo === Testing poly function definition error
cat > test-code.nelua <<\EOS
    ## for i=1,2 do
      local function #|'f'..i|#(): #[i == 1 and primtypes.integer or primtypes.number]#
        return 1
      end
    ## end
EOS
./nelua $nelua_args test-code.nelua

cat > test-code.nelua <<\EOS
    local function f1(): integer return 1 end
    local function f2(): number return 1 end
EOS
./nelua $nelua_args test-code.nelua

echo === Testing print types error
cat > test-code.nelua <<\EOS
    local n: float64
    local s: string
    local b: boolean
    local a: [2]int64
    local function f(a: int64, b: int64): (int64, int64) return 0,0 end
    local R: type = @record{a: integer, b: integer}
    function R:foo() return 1 end
    global R.v: integer = 1
    local r: R
    local tn = #[tostring(n.type)]#
    local ts = #[tostring(s.type)]#
    local tb = #[tostring(b.type)]#
    local ta = #[tostring(a.type)]#
    local tf = #[tostring(f.type)]#
    local tR = #[tostring(R.type)]#
    local tr = #[tostring(r.type)]#
EOS
./nelua $nelua_args test-code.nelua

cat > test-code.nelua <<\EOS
    local n: float64
    local s: string
    local b: boolean
    local a: [2]int64
    local function f(a: int64, b: int64): (int64, int64) return 0,0 end
    local R: type = @record{a: integer, b: integer}
    function R:foo() return 1 end
    global R.v: integer = 1
    local r: R
    local tn = 'float64'
    local ts = 'string'
    local tb = 'boolean'
    local ta = 'array(int64, 2)'
    local tf = 'function(a: int64, b: int64): (int64, int64)'
    local tR = 'type'
    local tr = 'R'
EOS
./nelua $nelua_args test-code.nelua

echo === Testing generate functions error
cat > test-code.nelua <<\EOS
    ## local function make_pow(N)
      local function #|'pow' .. N|#(x: integer)
        local r = 1
        ## for i=1,N do
          r = r*x
        ## end
        return r
      end
    ## end

    ##[[
    make_pow(2)
    make_pow(3)
    ]]
EOS
./nelua $nelua_args test-code.nelua

cat > test-code.nelua <<\EOS
    local function pow2(x: integer)
      local r = 1
      r = r * x
      r = r * x
      return r
    end
    local function pow3(x: integer)
      local r = 1
      r = r * x
      r = r * x
      r = r * x
      return r
    end
EOS
./nelua $nelua_args test-code.nelua

echo === Testing unpack ast nodes error
cat > test-code.nelua <<\EOS
    local function f(#[aster.unpack{aster.IdDecl{'x', aster.Id{'integer'}},
                                    aster.IdDecl{'y', aster.Id{'integer'}}}]#)
      print(x, y)
    end
    f(1, 2)
EOS
./nelua $nelua_args test-code.nelua

cat > test-code.nelua <<\EOS
    local function f(x: integer,
                     y: integer)
      print(x, y)
    end
    f(1, 2)
EOS
./nelua $nelua_args test-code.nelua

exit 1

echo === Testing require error
cat > test-code.nelua <<\EOS
require 'examples.helloworld'
EOS
./nelua $nelua_args test-code.nelua

cat > test-code.nelua <<\EOS
require 'examples/helloworld'
EOS
./nelua $nelua_args test-code.nelua

[mingodad]

Also would be nice if the make test could accept arguments to test only one (by number) test or only the failed tests.

[mingodad]

It seems to be something with memory (but valgirnd reports nothing) because sometimes make test-quick or make test pass.

[edubart]

I made recent changes so the test suite can pass with Lua 5.3 again, all tests succeeds with the following command for me:

LUA_PATH_5_3=".`/lualib/?.lua;./?.lua" lua5.3 spec/init.lua
505 successes / 0 failures / 8.705784 seconds

I am using unpatched lua 5.3, patching is not really needed if you have lpeglabel, hasher, lfs, chronos and term installed in luarocks.

Also would be nice if the make test could accept arguments to test only one (by number) test or only the failed tests.

This exists, use make test LESTER_FILTER="test name" for example, you can also export other environment variables for customizing tests, for more options read lester.lua.

It seems to be something with memory (but valgirnd reports nothing) because sometimes make test-quick or make test pass.

In my experience when this happens with Lua, it's usually due to undefined behavior, like using # in a table with nil holes, and this is exactly what happened. The function filter_ast_for_check was setting nil holes of a table, later stringfy_astnode was iterating the array part of the same table. Seems like while in Lua 5.4 filling nil holes of a table t doesn't change #t, in Lua 5.3 the behavior is random. I have fixed this in the latest commit to not fill a table while nil in the middle. This was only affecting the test suite not the language itself.

Let me know if this solves your issues.

[edubart]

More on the bug, by reading Lua 5.4 reference manual:

When t is a sequence, #t returns its only border, which corresponds to the intuitive notion of the length of the sequence. When t is not a sequence, #t can return any of its borders. (The exact one depends on details of the internal representation of the table, which in turn can depend on how the table was populated and the memory addresses of its non-numeric keys.)

I was aware of that and I remember doubling checking that, in Lua 5.4, if I populate a table with a known size upfront then creating nil holes later would not change the border of the table, thus #t would be kept fixed. But this is not true for Lua 5.3, more likely an implementation detail change, anyway the code does not depend anymore on that implementation detail behavior.

[mingodad]

Thank you ! While doing my experiments with LJS I made a shell script to generate Lua/LJS code from nelua sources in this repository and some do generate some code others emit a bunch of errors and some of the ones it generated aren't valid Lua/LJS code.

Maybe the output of this script can give ideas to improve nelua.

#check-generator.sh
for fn in `find . -name '*.nelua'`
do
    echo $fn
    #./nelua -g ljs -o $fn.ljs $fn
    ./nelua -g lua -o $fn.lua $fn
done

#check-generator-clean.sh

#for fn in `find . -name '*.nelua.ljs'`
#do
#    echo $fn
#    rm $fn
#done
for fn in `find . -name '*.nelua.lua'`
do
    echo $fn
    rm $fn
done

Output of lib/traits.nelua.lua:

-- Generated by Nelua 0.2.0-dev
-- Compile Hash: 3T3VnNBJzXkUBZbrfWasRTrUJDmz

traits = 
traits.typeid = 
traits.typeinfo = 
function traits.typeidof(v)
end
function traits.typeinfoof(v)
end
function type(v)
end
return traits

[edubart]

Thank you ! While doing my experiments with LJS I made a shell script to generate Lua/LJS code from nelua sources in this repository and some do generate some code others emit a bunch of errors and some of the ones it generated aren't valid Lua/LJS code.

The Lua generator is unmaintained and a hidden feature, it is not something Nelua advertises for, I even consider removing it to not mislead people.

Much of the Nelua standard library cannot compile back to Lua, as it uses pointers, passing by value semantics, calling C functions directly, things in lower level programming languages and not possible to compile to Lua, so of course using any Nelua standard library will not work out. Lua backend would need a completely different standard library just with type notations and less implementations. If you are looking for typed Lua check Teal, it add types notations and new syntax to Lua with just Lua backend in mind, while Nelua do everything with the C backend in mind. Also there is not enough flexibility in the compiler to plugin different backends (why do extra work for some unused feature and out of the goals?).

I talk a little more about the Lua backend here https://github.com/edubart/nelua-lang/discussions/189

[decalek]

I am not advocating the lua backend, because as you said, it is out of the project goals, but at some further stage a Lua+FFI one might be good for the PR:

Imagine something popular in Lua with thin C core (as Lite-XL for instance) translated to Nelua for the C part, as a demo of the language.

Then a build flag for targeting LuaJIT+FFI, instead of C - "Native development with hot-reloading" :-) (which will be part of the Zig marketing in the future) + an option for lightweight distribution, for the people which prefer things similar to the Turbo web framework.