Support for redefining the standard library

[tooolbox]

I'm using a Lua module that modifies string to add additional functions for handling utf8. I would like to write some Teal definition files so my linter doesn't complain when I use these functions.

global record string
    utf8lower: function(string): string
end

The above works in that Teal knows my new function is legit, but it blows away the normal definition of string, so I am now in a worse situation because all the normal methods are unknown. What I really need is something like this:

// utf8-things.d.tl
string.utf8thing = function(string): string

// or
global record extension string
    utf8lower: function(string): string
end

I read in the tutorial that the stdlib is declared as const, but I'm not sure the motivation for this or if it's just a matter of no syntax. Monkey-patching in Lua seems to be a thing, and it would be handy to be able to express this in Teal definition files. Not necessarily for crazy meta stuff, but for adding/replacing methods.

[hishamhm]

I read in the tutorial that the stdlib is declared as const, but I'm not sure the motivation for this

The generated code makes assumptions that depend on the standard library being constant: It automatically adds polyfills using compat-5.3 and local-izes some accesses in the process.

Monkey-patching implementations should work, but a big thing in statically-typed languages is that the types are, well, static :) - so changing the interfaces of records after the fact would break the assumptions made be the compiler.

Teal already is a bit lenient on what it allows for the sake of familiarity with Lua idioms (e.g defining methods in functions as function Foo.bar(...) after the record Foo definition).

Messing with the contents of the standard library tables isn't a great idea in general (think what happens if multiple libraries you depend on decide to do it in incompatible ways), so I don't think Teal should provide features to promote these patterns.

~For the time being, unfortunately if you really need to use a customized definition of standard library tables, you'll have to redefine the whole thing.~ For the time being, that remains not possible without patching Teal, but I'm keeping this issue open as a possible future feature.

[lenscas]

Though I agree that libraries shouldn't have a way to actually edit it. I do think there is benefit in teal being able to consume a .d.tl file and going "this is the std now". However, this should ONLY work for projects (so, be done through the tlconfig.lua file or similar) and not for libraries. As in, loading a library should NOT result in the std definitions being changed.

[tooolbox]

Messing with the contents of the standard library tables isn't a great idea in general (think what happens if multiple libraries you depend on decide to do it in incompatible ways), so I don't think Teal should provide features to promote these patterns.

For the time being, unfortunately if you really need to use a customized definition of standard library tables, you'll have to redefine the whole thing.

Well, I understand. Bit of a shame since there are libraries in the wild you might want to use that do this, but I see how it may not make the cut in terms of time investment vs. gain.

I wonder if it would work to redeclare string as a union of the existing string plus my own record with additional methods. That would solve it, I think. Might require some kind of type aliasing though which I don't see that Teal supports. Either way, I'm not in front of my dev machine to test at the moment, but perhaps later.

As in, loading a library should NOT result in the std definitions being changed.

Not sure why you'd want to prohibit a .d.tl from modifying the stdlib type definitions if it does in fact modify the actual stdlib. Seems like a footgun because linting becomes inaccurate without special knowledge and separate configuration in tlconfig.lua. Defend your position, good sir :wink:

[lenscas]

I want it so libraries can't modify the stdlib for the same reason that @hishamhm mentioned. What to do if 2 libraries get loaded that modify the stdlib in incompatible ways?

A .d.tl file that modifies the std should only be possible to be loaded through the tlconfig.lua file, every other way should ideally error. This way, if a user wants to work with libraries that alter the stdlib it is on them to define how those changes interact with each other.

Also, those changes are global, which means that even scripts that don't depend on this library are effected. This means that these kind of changes need a global way to be edited, not a local one.

[tooolbox]

I want it so libraries can't modify the stdlib

And yet, there are libraries that modify the stdlib. My perception of Teal's mission is to have typesafe Lua, including type definitions for existing Lua libraries. Some Lua libraries--I have no idea what percentage; I have run across a single one that I needed--modify the stdlib. One cannot wish this out of existence. So if you need such a library, Teal can't reflect the changes without blowing away the entire definition and redoing it yourself, which I think is a little onerous.

What to do if 2 libraries get loaded that modify the stdlib in incompatible ways?

I am not familiar with all the ways the stdlib might get munched. I'm sure you could do gross metatable inheritance monkey-patch rewriting nonsense that laughs at type safety, in which case yes, I would think Teal is off the table. But the one example of stdlib mods I have seen is adding a method. If I had two libraries that did that, and Teal had a way to express this in the type system, then my linter (and whatever eventual autocompletion) would reflect the final type with all its methods, which would be great!

Let's look at another example: I load two libraries that each overwrite the same method, with different signatures. If they both have .d.tl files and Teal evaluates all the types sequentially, then the latest type signature would be respected, which would reflect how the actual Lua would run. Then if you attempt to call that method with the former signature, you get a linting error and can trace down what's happening and see it's because of the collision. So again, this is all upside.

Maybe my relative inexperience with Lua is causing my confusion. I don't see how two modules could modify stdlib in an "incompatible" way that wouldn't in fact benefit from the ability to define those modifications in .d.tl such that the project author would be more likely to notice incompatibilites and bugs.

This way, if a user wants to work with libraries that alter the stdlib it is on them to define how those changes interact with each other.

I don't see why this would be the default. If I pull in a module that modifies string and another that modifies os they do not overlap. So then I have to write my own definition file? Better DX would be default-allow a module to modify stdlib with .d.tl and then if I specifically pull in two modules that both modify string to a sufficient degree that Teal can't make heads or tails of them, then I am responsible for writing my own definition file because I am choosing to use two incompatible modules.

Also, those changes are global, which means that even scripts that don't depend on this library are effected. This means that these kind of changes need a global way to be edited, not a local one.

Hm. This point does speak to me. But in the case of a library adding methods to stdlib, it seems like only your "main" script would use those features, so in that case local would be fine, no? Unless you have an example otherwise.

[lenscas]

Technically it is possible to write code that breaks when adding methods to a module. This is because modules in lua are just tables and thus can be given meta tables and can be iterated over. It is also possible to dynamically get functions out of them which could break if there are suddenly more methods.

I don't think that code is likely to exist but at this point nothing surprises me anymore.

Also, pretty sure that in the js world adding new methods to existing types is a big no no despite being possible. I feel the same is/should be true for lua and as a result don't see why teal should encourage it.

The only reason u like the idea of allowing people to define their own std is because lua is often embedded in software and having its std messed with for sandboxing or compatibility reasons.

[tooolbox]

Technically it is possible to write code that breaks when adding methods to a module. This is because modules in lua are just tables and thus can be given meta tables and can be iterated over. It is also possible to dynamically get functions out of them which could break if there are suddenly more methods.

Can you expand on this with an example? This would be because you're taking the nth method from the string table, I suppose? Hard to imagine anyone doing that in real-world code.

Also, pretty sure that in the js world adding new methods to existing types is a big no no despite being possible.

Not really. One huge example is that jQuery plugins add methods to the main jQuery object. In JS you can modify the prototype object or subclass, etc. I believe plain old monkey-patching is looked down upon except for emergencies when you need to bypass a bug, but the tools for extending objects are definitely put to use.

I feel the same is/should be true for lua and as a result don't see why teal should encourage it.

The fact remains that some libraries add methods to stdlib. Do you think that Teal's lack of support for this will cause such libraries to fade away? Seems unlikely. So then Teal just has a gap where real-world library usage can't be fully typechecked.

I am interested primarily in getting things done, and secondarily in having full and proper typechecking as a safety net. A mindset of discouraging me from using a particular type of library, when it may be the only one I can find that does a specific thing, comes across as high-handed.

One thing that the Go Team often does in development is analyze the existing corpus of Go code, when determining how to approach a particular problem. From that, I'm curious what percentage of Lua libraries patch stdlib and in what way. If I could know that I somehow ran across the single repo out of thousands that does this, it would be clear that it was a very unique case that didn't merit any effort from the Teal team.

Looking around further, I have found mention of embedding and/or intersection types. That should the specific use case I am running into. Since those are looking to be on the roadmap, I can probably get off my soapbox.

[uramer]

As a follow up to my question in https://github.com/teal-language/tl/issues/590:

It's very common for Lua to run in a sandbox, with a heavily customized API. So much so that different Lua environments are sometimes seen as their own dialects of Lua. In the majority of those cases, some parts of the standard library are not available.

I think it would be very useful to at least allow re-defining the standard library completely. As an example, in the project I'm working on, there is an explicit subset of the standard library documented: https://openmw.readthedocs.io/en/latest/reference/lua-scripting/overview.html

Only a limited list of allowed standard libraries can be used: coroutine, math (except math.randomseed – it is called by the engine on startup and not available from scripts), string, table, os (only os.date, os.difftime, os.time).

We also have LDT (.doclua) typed documentation for them, which I use to generate .d.tl files. I was very disappointed to find out I can't have correct autocomplete and typechecking (or rather lack of them) for, say, the debug library.

Another example would be Roblox's Luau, which also explicitly documents standard library limitations: https://luau-lang.org/sandbox

But realistically, almost any user-facing Lua environment will not expose the debug and os libraries at the very least.

[hishamhm]

I think it would be very useful to at least allow re-defining the standard library completely. As an example, in the project I'm working on, there is a explicit subset of the standard library documented: https://openmw.readthedocs.io/en/latest/reference/lua-scripting/overview.html

This is not currently supported, but it would be a welcome feature. (Hint to anyone thinking of implementing this: I assume it would also imply --gen-compat off, to avoid assumptions made by the Teal code generator.)

I was very disappointed to find out I can't have correct autocomplete and typechecking (or rather lack of them) for, say, the debug library.

Is it that disappointing that debug. autocompletes in the editor even though the function is not available in your environment?

I understand that environments that have incompatible builtin libraries would benefit from customized definitions overriding the standard library, but certainly having definitions for missing functions is not that big of a deal? It's not like they're breaking a sandbox or anything.

Another example would be Roblox's Luau

Luau is at this point its own language, and not a version of Lua — it is not a valid target for Teal, and it has its own static type system. But I get your point; there are many customized environments using vanilla Lua or LuaJIT that would benefit from this.

[uramer]

Is it that disappointing that debug. autocompletes in the editor even though the function is not available in your environment?

Of course it's not a critical issue, but I can see it being confusing for a newbie programmer, and I expect a decent amount of those using our API. Although I was mostly disappointed because I already had the declaration files, and no way to use them :)

Luau is at this point its own language

As far as I understand, Luau is mostly backwards compatible with Lua. But yeah, it is technically a different language.