Introduce 5.4 target with support for <close>

[euclidianAce]

• Adds a TargetMode of "5.4"
• Replaces Variable.is_const and Node.is_const with an Attribute enum consisting of "const" and "close"
• __close added as a metamethod
• breaking: tl.pretty_print_ast now takes a TargetMode to know when to emit \<close> attributes. Additionally it can now fail due to this. Currently the only failure that can happen is due to trying to emit something with a \<close> attribute for a non 5.4 target as this produces different behavior (as opposed to \<const> which can be statically verified and removed without breaking the program) TODO: This error is currently handled as a call to error, but it should probably be propagated to the user.
• FILE has a __close metamethod
• tl.init_env will return an error if gen_target is 5.4 and gen_compat is not explicitly off
• find_var returns the variable's attribute rather than a boolean
• add_var takes an Attribute rather than boolean for is_const
• added type_is_closable. add_global still takes a boolean for is_const since globals can never be \<close>
• type checker correctly validates that:
  • there can be only one \<close> variable per declaration (to ensure correct order of __close metamethods being called)
  • the target must be 5.4
  • the type of the variable is closable. We can do better with this as we could also prevent types that coerce to a closable annotation (mostly empty tables) since there are no literal \<close> values. (much like Ada's limited types)
  • globals can't be \<close>
  • \<close> variables can't be assigned to
• tl.proccess_string previously did not pass gen_target into its TypeCheckOptions. (TODO: It seems redundant to have gen_target and gen_compat in the TypeCheckOptions when Env also has those fields. Can they ever be different?)
• added tl.target_from_lua_version(string) to transform _VERSION into a TargetMode

All that is left is to add tests to verify that the above isn't a lie :)

[github-actions[bot]]

Teal Playground URL: https://506--teal-playground.netlify.app

[hishamhm]

Cross-referencing https://github.com/teal-language/tl/issues/252

The "Teal dialects" issue still crosses my mind... right now I'm wondering if it would make sense to have limited support for Lua versions < 5.4.

I'm not saying this is a requirement for this PR, but I'm going to jot some notes on how something like this could be done:

• emit explicit getmetatable(v).__close(v) calls for all <close> variables in scope before each break and return statement
  • there's an extra complexity for return <exp...> because the expression (which may be variadic!) needs to be evaluated before the __close calls (the Lua manual states that the presence of to-be-closed variables in scope makes returns never be a tail call, that's why). This is not too complicated to pull off using generated temporary variables, except for the case of variadic returns. Well, for variadic the easiest would be to create an auxiliary function tl_close_and_return(integer, ...) such that, if you need to close variables x and y and emit a return f(x), you'd instead emit return tl_close_and_return(2, x, y, f(x)). For LuaJIT efficiency, it would be probably best to make specialized versions for common cases, such as return tl_close_2_and_return(x, y, f(x)), which could pull off the operation without producing a temporary table.
• if an expression f in a for x in f do statement returns 4 value, then the resulting code for Lua 5.4 should be turned into a while form so that the fourth value can trigger a __close in the event of a break (and then yay! for file in lfs.dir() no longer leaks a descriptor if you break from the iteration!)

Essentially, that would make <close> work for LuaJIT and Lua < 5.4, except for error handling, where closing responsibilities would still be subject to the GC finalizer (as is the case for coroutines that are not resumed, even in 5.4).

[euclidianAce]

Just expanding on some stuff as I'm writing the code to validate this stuff:

emit explicit getmetatable(v).__close(v) calls for all <close> variables in scope before each break and return statement

And any goto that leaves the block ;) (and check for nil since local x <close> is a valid Lua 5.4 program)

Essentially, that would make work for LuaJIT and Lua < 5.4, except for error handling

To add to this, one big concern with errors is that __close metamethods are passed the error object in the case of an error. And errors in __close are protected against, making it very difficult to emulate.

But looking pragmatically, if you're relying on compatibility code being generated, you probably are just gonna copy your (hopefully idempotent) __gc method to __close as well? And destructors are such a nice feature to have I don't think people will mind that the behavior isn't 1-to-1 with 5.4. Just being able to do stuff like:

local function read_file(name: string): string, string
   local f <close>, err <const> = io.open(name, "r")
   if not f then return nil, err end
   return f:read("a")
end

(Writing that snippet made me finally realize why nil is a valid <close> value :laughing:)

Personally my take (as someone that deals with different versions of Lua in different projects) is that I am totally fine with locking features behind --gen-target flags, such as <close>, goto, etc. It could even help with the situation of "I'm running 5.3 right now, but want to target the subset of features that 5.1 has"

[lenscas]

Personally my take (as someone that deals with different versions of Lua in different projects) is that I am totally fine with locking features behind --gen-target flags, such as , goto, etc. It could even help with the situation of "I'm running 5.3 right now, but want to target the subset of features that 5.1 has"

that is how I feel about it as well, especially with luajit still being popular (which is pretty much lua 5.1). The compat library helps, but that is a C library if I remember correctly, which means that on most environments were I expect it to be needed it can't actually be used.

[hishamhm]

And any goto that leaves the block ;) (and check for nil since local x is a valid Lua 5.4 program)

Ah, all the flow typing analysis that is implemented already solemnly ignores goto... I didn't even bother to take it into consideration when doing a single-pass type checker. To do that properly, the right thing to do would really be to implement an IR with basic blocks, then goto would be supported for free along with while, for, if...

To add to this, one big concern with errors is that close metamethods are passed the error object in the case of an error. And errors in close are protected against, making it very difficult to emulate.

Good point! But it would still fall in the "except for error handling" caveat.

And destructors are such a nice feature to have I don't think people will mind that the behavior isn't 1-to-1 with 5.4.

Yeah, that's what made me do that whole thought exercise on how to implement it, and almost made me itchy to give it a try (perhaps after this PR is done?)

I am totally fine with locking features behind --gen-target flags, such as <close>, goto, etc. It could even help with the situation of "I'm running 5.3 right now, but want to target the subset of features that 5.1 has"

I totally get that feeling and LuaRocks is written like that, so it's probably over 10 years now that I've been walking on eggshells with that codebase to ensure everything works on every version. But my attempt with bundling compat tighly to Teal like that was really to try to get the Lua universe out of this 5.1-stasis. Especially since when we say "5.1" in 2022 we almost always mean "LuaJIT", and the "LuaJIT + Compat" combo is really much closer to 5.2+ than it is to vanilla 5.1.

that is how I feel about it as well, especially with luajit still being popular (which is pretty much lua 5.1). The compat library helps, but that is a C library if I remember correctly, which means that on most environments were I expect it to be needed it can't actually be used.

Compat is compatible with LuaJIT (which you could also call "sort of 5.1 but not really"), so if you're bundling a Lua VM written in C, you should be able to bundle it as well.

But if you're talking about things like scripting games that don't allow requiring additional modules, then yeah; but those often add their own restrictions to the standard library too (e.g. no os.execute()), so the Teal compiler just assumes a recent Lua stdlib available, and if you're compiling for an environment that doesn't have any parts of it, then it's up to you to avoid using missing functions (either because the game removed functions or because it's actually an older Lua version).

In short, I'd still rather keep the Lua-version-shenanigans away from Teal programmers as much as possible, but I'd like to find a good pragmatic middle-ground eventually.

Let's go with the approach @euclidianAce is taking in this PR now. At a later time, we could take a shot at a compat-<close> PR and see if it would be worth the effort.

[hishamhm]

@euclidianAce just checking back on this — since you added tests, what's missing for this to be ready for review/merge? I'm thinking of pulling in some code from #449 into master and it would be best to merge this first so that we don't end up with merge conflicts.

[euclidianAce]

@hishamhm From what I remember (as I haven't gotten the chance to look back at this) I wanted to basically do one final look through the code to make sure that errors were getting propagated correctly since this pr makes most if not every tl.* function fallible. So I wanted to make sure that any internal calls to these handle their errors nicely.

But after that I think it'd be good to merge

[euclidianAce]

After getting a chance to re-look this over, I think the places where errors are propagated make sense and think it's ready to merge (or at least be reviewed for merging :P)

[hishamhm]

I had already reviewed the major code changes before, so it already L'ed GTM before the tests, so if you're happy with the PR then I'm happy to merge :)