Codegen function from a tag string to support ReactPy's virtual DOM

jimbaker commented 2 years ago

We can avoid overhead of parsing a tag string on each usage by alternatively code generating a corresponding function, then memoizing it. I have implemented a gist to show how this can be done. This code implements an html tag for ReactPy:

Parses the tag string with HTMLParser from the stdlib, in a similar fashion as done in https://github.com/jimbaker/tagstr/issues/1.
Code generates a Python function. As part of this, it passes in the lambda-wrapped expressions to be interpolated as args. The signature of this function is compiled(vdom, /, *args), where vdom is the DOM constructing function. In this case, the vdom function in idom.core.vdom, and it is very similar to other tools like ViewDOM's VDOMNode constructor. I expect this will be a common pattern.
Calls the tag function as usual.
Memoizes the function for subsequent use. Note that lambda-wrapped expressions will change on each usage of the tag string, so we can use their underlying code object for the memoization key, which is constant.

For HTML without interpolation, this looks like the following. First, the example from the IDOM docs, followed by the equivalent with the new tag:

from reactpy import html
from reactpy_tag import html as html_tag

layout1 = html.div(
    html.h1("My Todo List"),
    html.ul(
        html.li("Build a cool new app"),
        html.li("Share it with the world!"),
    )
)

layout2 = html_tag"""
<div>
    <h1>My Todo List</h1>
    <ul>
        <li>Build a cool new app</li>
        <li>Share it with the world!</li>
    </ul>
</div>
    """

assert layout2 == layout1

Next I tried plugging this into a Hello, World example from ReactPy's intro:

from fastapi import FastAPI

from reactpy import component
from reactpy_tag import html
from reactpy.backend.fastapi import configure

a = 6
b = 7

@component
def HelloWorld():
    return html"""
    <h1>Ultimate answer: {a}*{b} is {a*b}</h1>
    """

app = FastAPI()
configure(app, HelloWorld)

Then assuming the above is in main.py, run with uvicorn main:app

This requires installing reactpy, fastapi, and uvicorn. So there's some setup required, but interesting to see how tag strings work with an interesting project.

jimbaker commented 2 years ago

I should mention that I used FastAPI/Uvicorn to run the web app because I couldn't get the dev web server to work on my development build from the tag string branch in https://github.com/jimbaker/tagstr/issues/1

jimbaker commented 2 years ago

As expected, implementing support for ViewDOM is straightforward. Here's the necessary diff - we just need to adapt for the minor calling convention difference between the vdom constructor in IDOM and VDOMNode constructor in ViewDOM:

$ diff vdomtag.py idomtag.py
10c10
< from viewdom import VDOMNode, render
---
> from idom.core.vdom import vdom, VdomDict
35c35
<         self.lines.append(f"{self.indent()}vdom('{tag}', {dict(attrs)!r}, [")
---
>         self.lines.append(f"{self.indent()}vdom('{tag}', {attrs!r}, ")
42c42
<         self.lines.append(f'{self.indent()}]){"," if self.tag_stack else ""}')
---
>         self.lines.append(f'{self.indent()}){"," if self.tag_stack else ""}')
100c100
<     return compiled(VDOMNode, *args)
---
>     return compiled(vdom, *args)

See https://gist.github.com/jimbaker/8ca34e920eb7245a0d1cc093fa8e91f0 for the full code.

jimbaker commented 2 years ago

So this almost works as expected:

from viewdom import render  # from https://github.com/pauleveritt/viewdom
from vdomtag import html  # from https://gist.github.com/jimbaker/8ca34e920eb7245a0d1cc093fa8e91f0

def Todo(prefix, label):
    return html'<li>{prefix}{label}</li>'

def TodoList(prefix, todos):
    return html'<ul>{[Todo(prefix, label) for label in todos]}</ul>'

b = html"""<html>
    <body attr=blah" yo={1}>
        {TodoList('High: ', ['Get milk', 'Change tires'])}
    </body>
</html>
"""

print(render(b))

resulting in

<html><body attr="blah&#34;" yo=""><ul><li>High: Get milk</li><li>High: Change tires</li></ul></body></html>

The problem is that the interpolation for the property yo= is not properly being put in the generated code. This should be a straightforward fix to (currently very naive) VdomCodeBuilder in the gist.

gvanrossum commented 2 years ago

Neat! I suspect trouble when the html parser buffers data -- I think if I call b.feed("x"); b.add_interpolation(i); b.feed("y") you might get the interpolations and constant text in the wrong order. But we can fix that in the parser if we desire. (You neatly traipsed around this in the demo by using {i} along with {j} as the data in the Gist, and by having no constant data at all in the demo above.)

jimbaker commented 2 years ago

Hah, these examples needs some proper development now. I too was surprised I could get away with what I did with HTMLParser :grin: I will put together a branch on this project so we can manage example tag libraries with requirements and tests - copying and pasting into gists is just too much.

One nice observation is that for DOM libraries like IDOM, it is possible to mix usage like html.h1('foo') with html"<h1>bar</h1>". (For ViewDOM, it needs to be using a different name.)

gvanrossum commented 2 years ago

No need for a branch. Just create a folder.

jimbaker commented 2 years ago

The problem with how we use html.parser and its interplay with interpolations should now be solved in https://github.com/jimbaker/tagstr/blob/main/examples/htmltaglib.py

Whether we should have a solution that depends on an internal part of html.parser, namely its rawdata attribute is another question. (Examples!) But in a nutshell, this attribute allows us to determine where in the parse we are - in a tag, and if so, what has been parsed up to that interpolation; or in the data portion of that tag. The internal goahead function should "handle data as far as reasonable", per its comment.

At some point, I will add proper tests.

gvanrossum commented 2 years ago

If this ends up in the language we should add a new API to htmllib to help us here. Until then we can depend on internals or clone htmllib and hack on it, whatever works. It's just a demo. If we can do it by depending on a few internals that's good enough.

rmorshea commented 2 years ago

The implementation from the Gist seems like something I wrote back when I was thinking of taking an approach similar to, but not quite the same as, pyxl - instead of parsing the HTML with HTMLParser I used htm, and instead of constructing a string you could compile() I built an AST you could exec directly.

jimbaker commented 2 years ago

The example code for this issue is now tracked in https://github.com/jimbaker/tagstr/blob/main/examples/htmltag.py

rmorshea commented 2 years ago

Another reference implementation I found using a PEG parser: https://github.com/michaeljones/packed

jimbaker commented 2 years ago

@rmorshea Given that we are targeting DSLs, it makes sense to have at least one example worked out for using a PEG parser like pyPEG or pyparsing.

gvanrossum commented 2 years ago

One trick that occurred to me when we're feeding the text into some kind of parser: Sometimes that parser may not be set up to handle interpolations (we already ran into this with the stdlib html parser). We could replace each interpolation with a special marker that includes the index of the interpolation (e.g. $$__1__$$) which is parsed as plain text, and then hunt for the markers in the resulting object tree and replace them with the corresponding interpolations.

pauleveritt commented 1 year ago

Jim's example includes the special marker. But there's still the open question about changing the parser internals. Do you think that's still on the table?

gvanrossum commented 1 year ago

I've honestly lost track what exactly this issue discusses, but if we wanted to add an API to the stdlib html parser to help it serve as a working example for this type of application I don't see why not. It would require someone (not me) to design that API and implement it, and some core dev (also not me) to approve it. I'm not sure if it would be required to spec this out in the PEP; my hunch is not (the PEP should focus on specifying the feature itself and providing a motivation so people understand how useful it might be). Honestly the stdlib html module seems a bit outdated, and there might be better 3rd party options available; we could see if one of those is interested in prototyping API support for tag strings.

pauleveritt commented 1 year ago

Let's close this ticket. I'm working on examples. I think I'll go in the direction of another example using a non-stdlib-htmlparser.

jimbaker / tagstr

Codegen function from a tag string to support ReactPy's virtual DOM #9