Too many local variables (limit is 200)

[allquixotic]

Hello,

I'm trying to get a fully featured RegEx library for Lua that will run in a pure Lua environment (no require, no FFI, no os or io packages, etc.) -- specifically the Lua 5.1 interpreter baked into Elder Scrolls Online.

For this I've set out to try to use onigurumajs, browserify, castl, and node-lua-distiller to get onigurumajs into a single Lua file with no dependencies.

Here's what I did, exactly (on macOS, but should work plus or minus a sudo on Linux, and probably even Windows Git Bash):

#!/bin/bash
set -e
rm -rf ooftest
mkdir -p ooftest
pushd ooftest
git clone https://github.com/bcoe/onigurumajs
git clone https://github.com/PaulBernier/castl
cd castl
git apply --ignore-whitespace << 'EOF'
diff --git a/lua/castl/constructor/date.lua b/lua/castl/constructor/date.lua
index e7faa74..302342e 100644
--- a/lua/castl/constructor/date.lua
+++ b/lua/castl/constructor/date.lua
@@ -92,29 +92,10 @@ end

 Date._timestamp = 0

-if luajit then
-    local ffi = require("ffi")
-    -- posix systems only
-    ffi.cdef[[
-        typedef struct timeval {
-          long tv_sec;
-          long tv_usec;
-        } timeval;
-        int gettimeofday(struct timeval *restrict tp, void *restrict tzp);
-    ]]
-
-    local te = ffi.new("timeval[1]")
-
-    Date.now = function(this)
-        ffi.C.gettimeofday(te, nil);
-        return tonumber(te[0].tv_sec * 1000 + te[0].tv_usec / 1000);
-    end
-else
     Date.now = function(this)
         -- TODO: write a C function to get milliseconds
         return time() * 1000
     end
-end

 Date.parse = function(this, str)
     -- TODO: parse RFC2822 only for now
diff --git a/lua/castl/modules/dkjson.lua b/lua/castl/modules/dkjson.lua
index a1c7c69..1468c9e 100644
--- a/lua/castl/modules/dkjson.lua
+++ b/lua/castl/modules/dkjson.lua
@@ -69,7 +69,7 @@ local _ENV = nil -- blocking globals in Lua 5.2
 pcall (function()
     -- Enable access to blocked metatables.
     -- Don't worry, this module doesn't change anything in them.
-    local debmeta = require "debug".getmetatable
+    -- local debmeta = require "debug".getmetatable
     if debmeta then getmetatable = debmeta end
 end)

EOF
cd ../onigurumajs
npm install -g browserify castl luamin npm-check-updates coffeescript
ncu xregexp -u
npm install
npm install lua-distiller
browserify index.js --bare > oniguruma.js
cp -rf ../castl/lua/castl .
castl oniguruma.js --babel --mini -o oniguruma.lua
./node_modules/lua-distiller/bin/lua-distiller.coffee -i oniguruma.lua -o oniguruma-all.lua
luamin -f oniguruma-all.lua  > oniguruma-all-min.lua
lua -i oniguruma-all-min.lua

The error:

$ lua -i oniguruma-all-min.lua
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
lua: oniguruma-all-min.lua:1: too many local variables (limit is 200) in function at line 1 near ','

or, unminified:

$ lua -i oniguruma-all.lua
Lua 5.3.5  Copyright (C) 1994-2018 Lua.org, PUC-Rio
lua: oniguruma-all.lua:25330: too many local variables (limit is 200) in function at line 25329 near ','

Line 25330 is basically a bunch of variable declarations inside the lodash library that is required by onigurumajs. The number of variable declarations definitely exceeds 200, hence the error.

Notes about stuff in the script:

• The castl patch is necessary both because lua-distiller doesn't find the ffi and debug modules (even when running fully privileged lua 5.3.5 locally), and because those modules aren't available in Elder Scrolls Online's extremely locked down lua interpreter anyway.
• Upgrading to xregexp 4.x using ncu was necessary to resolve a problem with browserify being unable to browserify the xregexp addons in version 3.x which oniguruma was asking for.
• I've tried this both with, and without minification, both on the castl side and the lua side.

There may still be deeper errors once these are resolved, but the issue with 200 variables must be resolved first.

Somehow your Lua code generator needs to detect >200 local variables in a scope, and stuff them into a table, then rewrite all the uses of them to use a table instead of direct local variable references. That sounds like a non-trivial effort, but anything that includes the lodash library, for starters, can't be used with castl as-is.

FWIW, tessel has the same problem :( https://github.com/tessel/t1-runtime/issues/658

I will be "happy" (as far as castl is concerned) if the transpiled Lua code works with a desktop Lua interpreter, e.g. the reference implementation as available on MacOS or Linux. But I have a feeling I will hit a few more issues based on the limitations of the Havok Script Lua engine of ESO, which you can read more about here in case you're interested: https://wiki.esoui.com/Esolua

[PaulBernier]

Hello Sean, thank you for taking the time to report this issue and writing a detailed description of it, I really appreciate this kind of contribution to the open-source ecosystem. Unfortunately CASTL was an experimental software I built for a start-up and I'm not actively maintaining it (besides trivial bugs). I actually remember some occurrences of this limitation and I think your proposed solution is reasonable but I'm sorry I won't have time to work on it (+ I haven't written any Lua for years).

I still had a look at your problem and there may be at least one thing you could give a try: I noticed that onigurumajs is only using a single function from lodash (omit) what you could do is modify (patch like your did with castl) onigurumajs to directly use that smaller package lodash.omit which may help avoid this 200 local variables error.

Best of luck.

[Olical]

So I hit this with https://github.com/Olical/cljs-lua-experiment too. I was trying to compile a ClojureScript program that could compile all other ClojureScript programs to it though which is probably asking a bit too much. This is so that I can basically drive Neovim (which has LuaJIT built in) with ClojureScript in a kind of native way. Scary, right!

I am considering having a play with castl one day to see if I can break things up into deeper and deeper closures or something as the local variable limit is hit... kind of a hack but it might work! If I have a play with that concept I'll link it here. Thanks for your existing work, it's a great starting place for many people! :tada:

[PaulBernier]

Thanks for sharing @Olical! Wish you the best of luck with your experiment ;)