search

teal-language / tl

The compiler for Teal, a typed dialect of Lua
MIT License
2.11k stars 109 forks source link

Proposal: TLX #519

Closed tooolbox closed 2 years ago

tooolbox commented 2 years ago

If Teal is to Lua as Typescript is to Javascript, .tl is to .ts, .d.tl is to .d.ts, but what is the Teal equivalent of .tsx?

txs files are just ts files with an XML-like syntax that transforms into straight ts. tsc knows how to typecheck it. It allows for building modularized chunks of markup in a typesafe manner.

Sample:

<Component name="foo">
  <h1>Hello World</h1>
</Component>

Under typical circumstances, transpiles to (very) roughly:

React.createElement("Component", {name: "foo"}, 
    React.createElement("h1", null, "Hello World")
)

So what if the Teal compiler could convert that same markup to (roughly) the following:

Component {
    name = "foo"
} (h1 {} "Hello World")
-- i.e.
Component({name = "foo"}, h1({}, "Hello World"))

In practical usage, you might do something like this:

local html = require('html')

local myTemplate = <html.div>
    <html.p>paragraph</html.p>
    <html.a href="#anchor">Text</html.a>
    <html.img src="file.jpg" class="big" />
</html.div>

-- or:

local myTemplate = html(function(_ENV)
    return <div>
        <p>paragraph</p>
        <a href="#anchor">Text</a>
        <img src="file.jpg" class="big" />
    </div>
end)

(Examples converted from here.)

The exact implementation of html or whatever library is actually irrelevant, it's all about the transform from XML to Lua, or actually Teal first so that typechecking can occur.

local count_display = function(count: number)
    return <div>{ count }</div>
end

local value = "5"
local page = <html>
    <body>
        <count_display count=value /> -- type error: expected number, found string
    </body>
</html>

Consider this a rough proposal sketch. I'm curious what the maintainers and/or community think.

lenscas commented 2 years ago

tsx files are typescripts answer to jsx files. And though it seems like it has support for some frameworks like react built in, for the majority of frameworks it will have to spit out jsx files.

Lua as far as I know doesn't have an equivalent to jsx files, so I personally see no reason why teal needs to introduce a tsx file equivalent.

The exact implementation of html or whatever library is actually irrelevant, it's all about the transform from XML to Lua, or actually Teal first so that typechecking can occur.

The implementation of such a library is important to typescript, as seen by the fact that it has only native support for react and for everything else it emits jsx files. Why would this be different for teal?

hishamhm commented 2 years ago

I agree with @lenscas 's assessment. This sounds like something to be handled at a framework level, and as far as I know no Lua framework offers anything similar. If a Lua web framework wants to implement an XML-to-Lua format in the vein of jsx, and they need something from us to offer improved Teal support, we'll be happy to discuss, but I don't see us spearheading that effort. Hope that makes sense!

tooolbox commented 2 years ago

tsx files are typescripts answer to jsx files. And though it seems like it has support for some frameworks like react built in, for the majority of frameworks it will have to spit out jsx files.

Yes, .tsx is typed .jsx, but tsc doesn't produce JSX, it produces raw JS. You configure the compiler to tell it what function calls you want. It can be the default of React.createElement() or it can be foo.bar(), whatever you like.

Lua as far as I know doesn't have an equivalent to jsx files, so I personally see no reason why teal needs to introduce a tsx file equivalent.

What you say is true and I understand the perspective; I'm just coming at it from a different angle.

I think that a "JSX for Lua" would be really cool and useful. I see libraries around that use Lua's parentheses-less function call syntax to declaratively build HTML, but it's a very different sensation from working with actual tags and attributes. It feels foreign and half-baked.

"JSX for Lua" would be neat, but would never succeed by itself because Lua has no culture of pre-processors in the way JS does. People just want to run their Lua. (The exception being people who use Teal, because the benefits are worth it, but even then it's implemented in Lua and allows you to still run .tl modules in a Lua VM with no separate transpile step.)

So, it's not that I'm saying Teal is obligated to have a .tsx equivalent, I'm saying that Teal's existence provides the opportunity to create a .jsx/.tsx equivalent for Lua, where it would otherwise never exist.

This sounds like something to be handled at a framework level, and as far as I know no Lua framework offers anything similar. If a Lua web framework wants to implement an XML-to-Lua format in the vein of jsx, and they need something from us to offer improved Teal support, we'll be happy to discuss, but I don't see us spearheading that effort.

To me, this is above the framework level. The very fact that JSX exists with a defined transform from markup to function calls allows arbitrary libraries to design their APIs to work with JSX. React, Preact, Vue, Inferno, Mithril, Hyperapp, Solid, the list of libraries that can use JSX goes on and on. (In terms of how many Lua libraries could use a feature like this, HTML generation is an easy one and I have ideas on others.)

Part of why I think Teal integration is important is for linting. If I write a theoretical "LX" tool that transforms .luax files to straight .lua, Teal will not be able to lint the XML gibberish I'm typing into my editor. If I make the linter run LX and then Teal consecutively, I will get valid warnings but the line/char positions won't match up for underlining/highlighting.

If we could defer, for a moment, the question of who is spearheading what, I'm curious if it would be looked on as a problem for Teal to have this capability? If I took whatever required time and implemented this and sent a PR, what would be the response and thought process behind that?

lenscas commented 2 years ago

Yes, .tsx is typed .jsx, but tsc doesn't produce JSX, it produces raw JS. You configure the compiler to tell it what function calls you want. It can be the default of React.createElement() or it can be foo.bar(), whatever you like.

Going by the documentation, only react is supported and for the rest it will have to produce jsx files.

From the documentation

Controls how JSX constructs are emitted in JavaScript files. This only affects output of JS files that started in .tsx files. react: Emit .js files with JSX changed to the equivalent React.createElement calls react-jsx: Emit .js files with the JSX changed to _jsx calls react-jsxdev: Emit .js files with the JSX changed to _jsx calls preserve: Emit .jsx files with the JSX unchanged react-native: Emit .js files with the JSX unchanged

https://www.typescriptlang.org/tsconfig#jsx

To me, this is above the framework level.

In JS it started at the framework level though. With frameworks like react adding ways to consume these jsx files and produce normal js files. Granted, they used compiler options by stuff like babel to do so, however there is nothing preventing people from making similar wrappers for lua if this is desired.

Typescript did not create the tsx files and forced JS to follow. I see no reason why in teal's case it should be different.

The very fact that JSX exists with a defined transform from markup to function calls allows arbitrary libraries to design their APIs to work with JSX. React, Preact, Vue, Inferno, Mithril, Hyperapp, Solid, the list of libraries that can use JSX goes on and on. (In terms of how many Lua libraries could use a feature like this, HTML generation is an easy one and I have ideas on others.)

I will admit that there is a use case for being able to transform code or even better yet define a new DSL and write it inline but I don't think that tsx/jsx is a good way to do this for Teal.

jsx/tsx works great for the JS ecosystem as it is really intertwined with html. So, it makes sense to give that special treatment. Lua on the other hand is not that intertwined with HTML nor XML. So giving those formats special treatment in teal does not make much sense in my mind. There are also other ways of being able to define DSL's and being able to write them inline.

For a very powerful example look at Rust's procedural macros. These are probably a bit too powerful to introduce to Teal but some crates already use them for great effect, yew with html! macro, mlua with its chunk! macro and many more.

There is also F# with its computation expressions. These are a bit more limited in what they can do but still allowed someone to make a proof of concept for getting inline assembly to work and though its version of inline html is far away from looking like html, it is still not too bad https://twitter.com/fsbolero/status/1517602581895434240

C# also tips it toes into this space and does so in its own way with linq expressions which efcore uses quite effectively for its sql DSL.

And if proc macro's are a bit too powerful (lets be honest, they are) Rust also has your back with its macro_rules! macro's.

So, to me it sounds like there are A LOT of options to give Teal users the ability to embed their own domain specific languages to their teal code and from those JSX feels to me the most limited as it both requires a specific file extension and only 1 transformation can be active at a time per project. Even worse: It is very much aimed at xml/html.

I think that if Teal should get such a system that it can and deserves to have a system that is less restrictive than that.

Sorry if this was a bit of a long one, have somewhat of a TL;DR: JSX/TSX solve the problem of being able to define a DSL as a user. There are a lot of ways that languages do this and JSX also has the most restrictions. I think we can strike a better balance for Teal, especially as some of the restrictions of JSX don't make a lot of sense for Teal.

tooolbox commented 2 years ago

Going by the documentation, only react is supported and for the rest it will have to produce jsx files.

The docs are unfortunately unclear in that section. Please scroll down and have a look at the jsxFactory setting, which does what I'm talking about. React was the forerunner, but it is a fact that tsc can map JSX to arbitrary function calls. Esbuild works the same way.

In JS it started at the framework level though. With frameworks like react adding ways to consume these jsx files and produce normal js files. Granted, they used compiler options by stuff like babel to do so, however there is nothing preventing people from making similar wrappers for lua if this is desired.

Yes, JSX started with React. For reasons why direct Teal integration is desirable rather than a pipeline of two tools, see the end of my previous comment. I suppose if Teal had something like sourcemaps, that would also solve the problem.

Typescript did not create the tsx files and forced JS to follow. I see no reason why in teal's case it should be different.

Please see my last comment about why Lua is different from JS. It's not about forcing anything. Teal's existence as a preprocessor for Lua creates an opportunity, that's all.

jsx/tsx works great for the JS ecosystem as it is really intertwined with html. So, it makes sense to give that special treatment. Lua on the other hand is not that intertwined with HTML nor XML. So giving those formats special treatment in teal does not make much sense in my mind.

That is fair. Lua currently does not have a big focus on HTML/XML, whereas it's very innate to JS.

I suspect there is a little bit of a chicken and egg issue. Lua isn't that used for web stuff, so why support .tlx? But supporting .tlx would probably make it more attractive for web things.

There are also other ways of being able to define DSL's and being able to write them inline.

That is an interesting list. I didn't look at everything you linked at, but I get your point.

Sorry if this was a bit of a long one, have somewhat of a TL;DR: JSX/TSX solve the problem of being able to define a DSL as a user. There are a lot of ways that languages do this and JSX also has the most restrictions. I think we can strike a better balance for Teal, especially as some of the restrictions of JSX don't make a lot of sense for Teal.

I think it's inaccurate to view JSX as a general DSL solution. It is a sort of DSL that lets you write function calls in the form of markup. The primary use case is generating markup (statically or dynamically). The reason that's the primary use case is because the source code gives you a preview of the final output, since it resembles it. It makes it easier to reason about and visualize.

That said, your point is that you want a general DSL solution. Is that necessarily mutually exclusive with support for JSX-like transforms?

If Teal is solely meant to be a typechecker, then ~JSX and a general DSL are out of scope, and I suppose I would need to look into the subject of sourcemaps and how that might apply to Teal for integration with a theoretical secondary processor.

lenscas commented 2 years ago

I suspect there is a little bit of a chicken and egg issue. Lua isn't that used for web stuff, so why support .tlx? But supporting .tlx would probably make it more attractive for web things.

There is A LOT more needed to get lua big in web stuff. WASM still has quite a bit of overhead due to high FFI costs. So, Lua will be slower at runtime compared to JS. Using Lua instead of JS also means the lua VM needs to be shipped to the client, which means that more data needs to be send to the client.

Both of these make using Lua as a scripting language in the browser impractical.

On the server side it could work but competition there is tense (Ruby on rails, JS/TS, C#, Python are all pretty big players here). It would be hard for Lua to make a significant dent. Also, interestingly enough most of those language do not have a jsx like system. They have a templating system, sure but that is far away from JSX.

That said, your point is that you want a general DSL solution. Is that necessarily mutually exclusive with support for JSX-like transforms?

A generic DSL solution would (if powerful enough) enable jsx like syntax without it needing it to become its own feature. So, to me they are mutually exclusive.

Perhaps it is worth it to start a discussion about different ways to add DSL support to see which (if any) is right for Teal as there are many ways of doing it and all have very wildly different benefits and drawbacks.

tooolbox commented 2 years ago

There is A LOT more needed to get lua big in web stuff. WASM still has quite a bit of overhead due to high FFI costs. So, Lua will be slower at runtime compared to JS. Using Lua instead of JS also means the lua VM needs to be shipped to the client, which means that more data needs to be send to the client.

Yes, and there is Fengari but as you say it's true you have to ship the VM to the client one way or another. I mean, you could cache it, but the browser would still have to load it up each time. I could more see a full-stack/isomorphic Lua framework that emits JS for the logic that will run in the client (there is a TS-to-Lua project extant, so not impossible to go the other way). But I am in fever dream territory, here.

On the server side it could work but competition there is tense (Ruby on rails, JS/TS, C#, Python are all pretty big players here). It would be hard for Lua to make a significant dent.

Yes, there's a lot of competition. Lua on the server has a pretty tiny market share, all things considered. I think it's used more for games than web. On that note, I wonder if ~JSX in Lua would be useful for layout of immediate-mode UIs.

Also, interestingly enough most of those language do not have a jsx like system. They have a templating system, sure but that is far away from JSX.

For sure. Hence my interest. Having used .tsx and experienced typechecked componentized UI, it's very hard to go back to a plain old templating system. Imagine having used Teal or TS and returning to Lua or JS. There's a repeated sensation while building and debugging templates that your editor should have warned you about a mistake you made, or been able to give you typing completions for parameters.

Why not then just build typesafe HTML templating for language X? I believe I saw a Rust library that does this. Well again, those languages don't have something like JSX, and many of them don't have a macro system that could accommodate it natively if you wanted to, but many of them have Lua VMs with interop. You could do your HTML generation in Lua, with Teal you have type safety, add ~JSX support in Teal and you are there--for many languages. I'm sure you lose some measurable amount of performance, but caching is a thing, and I think the DX lift would be worth it.

A generic DSL solution would (if powerful enough) enable jsx like syntax without it needing it to become its own feature. So, to me they are mutually exclusive.

I understand your perspective. Well, a generic DSL solution has its benefits and drawbacks, and I will leave that to you.

At this point I don't see myself convincing much of anyone along this route, but I appreciate the thoughtful discourse @lenscas

lenscas commented 2 years ago

Actually, a good while ago there was some talk about being able to define build scripts. Though it was dropped as part of tl it is something that cyan could adopt.

If it has those it is not unreasonable to make a build script that does what you want. Though, it requires parsing the .tlx (?is that the name we are going for?) files manually and spitting out .tl files that then get picked up by cyan/tl.

I don't know the current state of build scripts or cyan at all though. But if you want .tsx/.jsx kind of files in Teal that is probably the easiest way.

Going this route for every DSL may also not be a bad idea for Teal as it does not add new syntax to Teal or Teal files. This means that the language itself stays very close to Lua at the cost of a bigger build system (Which may not be a problem considering that make is a popular build system anyway).

I can see myself porting tealsql-cli to a system like this, if nothing else as just a proof of concept.

There are however 2 big problems with this:

Teal's internal abstract syntax tree is not stable yet and thus not exposed. So even files that are close to Teal's syntax can't really reuse parts of Teal itself.

The other problem is with tooling. Though this works great for working with files that already have tooling (like how tealsql-cli consumes .sql files) it will work out of the box, for custom formats like .tlx files custom tooling would need to be created.

This is something that I really should've thought and considered earlier. Sorry about that :sweat_smile:

tooolbox commented 2 years ago

Though, it requires parsing the .tlx (?is that the name we are going for?) files manually and spitting out .tl files that then get picked up by cyan/tl.

Yes, .tlx was what I was thinking.

I don't know the current state of build scripts or cyan at all though. But if you want .tsx/.jsx kind of files in Teal that is probably the easiest way.

Mmm, well I suppose I could look into build scripts, but I'm not sure how that solves the linting issue, which is the more valuable aspect from my perspective. It's like the esbuild philosophy; run tsc in your editor so you get typechecking while developing, whereas esbuild strips types with no checks at build time.

Teal's internal abstract syntax tree is not stable yet and thus not exposed. So even files that are close to Teal's syntax can't really reuse parts of Teal itself.

The other problem is with tooling. Though this works great for working with files that already have tooling (like how tealsql-cli consumes .sql files) it will work out of the box, for custom formats like .tlx files custom tooling would need to be created.

Agreed. If I make my own tool, I still fundamentally need a Teal parser, just with extra bits. As you say, Teal doesn't expose the lexer/parser/AST in any way, so I'm either cloning and modifying or re-creating it from scratch. And then there's two compiler passes, once to strip the XML and secondly to typecheck and strip types. Which, in fairness, may still be performant enough.

I could possibly get around the lack of sourcemaps by making the .tlx processor keep everything on its same line. Then when Teal does linting, the squigglies will at least land on the right lines, even if the character positions are garbage.

lenscas commented 2 years ago

Mmm, well I suppose I could look into build scripts, but I'm not sure how that solves the linting issue, which is the more valuable aspect from my perspective

Editor support is a problem regardless of how .tlx, or any way to write custom DSL's for that matter is implemented. With the custom parser feature I talked about it at least becomes possible for people to write their own tooling for editor support and easily decide when to run based on the file extension.

Agreed. If I make my own tool, I still fundamentally need a Teal parser, just with extra bits. As you say, Teal doesn't expose the lexer/parser/AST in any way, so I'm either cloning and modifying or re-creating it from scratch. And then there's two compiler passes, once to strip the XML and secondly to typecheck and strip types. Which, in fairness, may still be performant enough.

I can see a world where Cyan/Teal do a lot of heavy lifting. For example in a world where these custom parsers are I can see them being implemented in a way that these parsers return some kind of token tree rather than writing teal code directly. I can also see Cyan/Teal having ways to set custom spawns, error messages, etc so that the errors keep making sense.

Of course, that depends on this custom parser thing being a feature that gets included into Cyan in the first place but that is a discussion that deserves its own place rather than this feature request issue.

I'm going to see if I can do a nice writeup of this all in a bit and then make a new post about DSL's in general in the discussions tab to see what other peoples thoughts are. As I think supporting something like that is a good feature for languages to have, I just don't like how narrow this proposal is. However I also think that it should be possible to find a way that makes everyone happy :)

lenscas commented 2 years ago

@tooolbox made the post over at https://github.com/teal-language/tl/discussions/520

Feel free to throw your opinion in :)

tooolbox commented 7 months ago

Somebody made this for straight Lua: https://bvisness.me/luax/

grifx commented 6 months ago

This feature would make Teal more appealing. Tlx would be very useful in a wide range of applications: HTML pages, emails, PDFs, Word docs.

A pragmatic approach could be to include it directly in the compiler, which will boost Teal's popularity, and then extract it into a complex plugin system later on, if it ever exists.