search

aquametalabs / aquameta

Web development platform built entirely in PostgreSQL
GNU General Public License v3.0
1.1k stars 52 forks source link

widget: refactor to make es6 modules core #189

Open erichanson opened 5 years ago

erichanson commented 5 years ago

Huzzah. ES6 modules are close enough to being supported to totally rethink widgets. Currently the post_js field is just javascript that get turned into a dynamically created function, which is executed once the widget's HTML and CSS are on the page. We could easily just include the Javascript in the widgets HTML with a <script type="module"> tag, but it'd be cooler if Javascript was broken out from the HTML as it is now. Figure out how to make post_js be a module. Potentially add UI that generates any import code automatically and add immutable code lines to the top of the js content. Discuss.

erichanson commented 5 years ago

Sub-goal: Take another crack at server-side rendering of widgets.

There was a first attempt at this, which evolved into just making server-side-rendered templates.

But the dream is alive! -- Modular user interface components that can be rendered sensibly both on the server or the client. Here's some "I wish it worked like this" code.

Page Template

This template bootstraps a page. It is stored in the content field of a row in endpoint.template. It is rendered server-side. The widget() function is automatically included in plv8. If we get things right, this could be a widget too, one that would (presumably) only be rendered server-side.

<html>
    <body>
        <!-- call to render the main widget -->
        {{= widget('com.myproject.www:main') }}
    </body>
</html>

main Widget

Right now the html and css templates are rendered first using doT.js. They're then injected into the document at runtime. Finally, the post_js code is executed, with the name and id variables and the widget and endpoint objects brought into scope.

With this pattern, all calls to the REST API aka database calls are executed post-rendering. This is what causes the "big bubble up" style rendering of a typical Aquameta app, where things (visibly) pop onto the screen as the DOM is built-up at runtime. This is to be avoided. It's a pattern that still might be useful at times, but right now there is no way to do pure server-side rendering that uses server-side database calls. So, make it so.

To pull this off we need a place for code that gets run before the html template is rendered, and can add variables to the render scope. It can make database calls for example, and make, say, a AQ.RowSet available.

var orders = endpoint.schema('biz').table('order').rows({ where: ... });
<ul>
    {{~ orders }}
        {{= widget('order', { order: it?? })
    {{~}} <!-- does this even work?  how do we iterate over a rowset? -->
</ul>

So, we have two dimensions of context here:

  1. Where the code is run? (server, browser, or either one)
  2. When the code is run? (before or after the html template is rendered)
  server_js client_js common_js
pre_js ? ? endpoint.schema('foo').table('bar').rows()
post_js ? attach a click handler ?

So we have six (6) different Javascript code boxes? Depending on whether a widget is invoked client-side as dynamic HTML or server-side, we execute four (4) of those code boxes in each pipeline, always common_js and either server_js or client_js. Hmm.

IDE

The IDE could present the user with a choice of rendering schemes -- client-only, server-only, or both. Under the hood these would all be the same type of widgets, but it would affect the UI so users aren't overwhelmed with all these different places in the pipeline where Javascript can sensibly be executed.

micburks commented 5 years ago

Really great place to start! And thanks for making this, I spent quite some time having this discussion in my head.

I think trying to make SSR/modules work in the current widget world will lead to a super heavy framework that tries to do too much. So I'll try to work from the bottom up in this discussion.

I'm not sure I can fully construct my entire thought process with how I rewrote datum/widget, but I'll try to give a few reasons for the decision I made.

Modules

A js module gets on the page. Let's skip past how this happens because there are so many ways to do it - it's more important how that module operates.

Once a module is running on the client, what I want is for that module to be able to execute any code it wants. This means directly referencing another module

import something from 'something-in-the-database';

The something-in-the-database part of that import is a request that goes directly back to the server. Whatever is returned from the server, is executed by the browser as a module (we don't have the opportunity to do any magic on the client). So building off of how modules work (if it is important to allow them to work according to the spec, which I believe it is), the server either needs to do some on-demand code generation to return usable code, or the referenced field in the database is already in a JS module.

Without allowing native import, we would both be drifting from the new web standard and forcing ourselves to re-implement module loading (like so many sorry souls before us).

For reference, the legacy widget would make an AJAX request for a row in the database, wrap the post_js in a couple dynamic functions (passing in deps, inputs, and globals for the function) and evaling the whole thing.

So now (if we appreciate that code in the database should be native JS module code), we can think about how to bootstrap an application.

Templates

I personally (and I won't start this discussion here) am not sold on using a template engine. It gets the job done on initial render, but then you have to fill in the "update" cracks with all this imperative jquery code to sync state to DOM state. Data changes all the time (user input, server events, fetching data, fetching components, etc.), this becomes a nightmare very quickly with HTML templates.

Widget as UI Framework

When it comes down to it, I found that widget needs to either not do anything (allow import, then let the use decide how they want to render the DOM) or do [almost] everything. Doing everything would include providing a framework to abstract the DOM and providing primitives to represent a UI and how it connects with data in the database. Perhaps this would look more like Vue templates where DOM attributes have significance outside of HTML.

<li sync-rows="endpoint.session">{{row.id}}</li>

just conceptual.

I'll refer to these two options as "Framework that does nothing" and "Framework that does everything".

Server-Side Rendering

Framework that does nothing

I didn't start here because there is probably no clean solution. SSR with widget code that supports modules needs to run untrusted code in Node. Period. This is fine if you only run code from a trusted source, but is a horrible security flaw that could never be overcome otherwise. So far, we haven't proven the need to run OPC (other people's code), so I still like the possibilities with this idea.

Framework that does everything

A framework that abstracts DOM from the user could construct the HTML document quite simply. The only JS the user would write are event handles that can be bound on the client.

Data fetching

right now there is no way to do pure server-side rendering that uses server-side database calls

This was a big reason for me to rewrite datum to run in both the browser and node. I think whatever widget looks like, the datum layer needs to run in all environments.

With both of the options I outlined, data fetching in SSR is possible. This requires using the same datum layer on the client as on the server.

Conclusion

This is less structured than I wanted it to be. Hopefully I made a little sense.

I think the main focus should be to identify what the ideal widget solution looks like. Does "untrusted" code run in node/plv8? What level of DOM manipulation do users need to do? How do users use code from the community? Does the user bring their own tools (vdom rendering, etc.) or is everything provided for them? Can users do both? Ooh... power users can use a library, end-users can use a visual programming interface?

From what I've already built, I think server-side data fetching could fall into place for most solutions. The only issue would be plv8 versus node. I'm definitely weary of trying to make plv8 somehow work with modules. To be honest, I don't see the value add for putting framework code in a PL lang.

Modern JS frameworks (and honestly, something like ClojureScript) are really good examples of ways to solve the UI space.

erichanson commented 5 years ago

I think trying to make SSR/modules work in the current widget world will lead to a super heavy framework that tries to do too much. So I'll try to work from the bottom up in this discussion.

link

Modules

A js module gets on the page. Let's skip past how this happens because there are so many ways to do it - it's more important how that module operates.

Once a module is running on the client, what I want is for that module to be able to execute any code it wants. This means directly referencing another module

import something from 'something-in-the-database';

The something-in-the-database part of that import is a request that goes directly back to the server. Whatever is returned from the server, is executed by the browser as a module (we don't have the opportunity to do any magic on the client). So building off of how modules work (if it is important to allow them to work according to the spec, which I believe it is), the server either needs to do some on-demand code generation to return usable code, or the referenced field in the database is already in a JS module.

Without allowing native import, we would both be drifting from the new web standard and forcing ourselves to re-implement module loading (like so many sorry souls before us).

100% on board with supporting native import. Accept no substitutes.

For reference, the legacy widget would make an AJAX request for a row in the database, wrap the post_js in a couple dynamic functions (passing in deps, inputs, and globals for the function) and evaling the whole thing.

So now (if we appreciate that code in the database should be native JS module code), we can think about how to bootstrap an application.

Templates

I personally (and I won't start this discussion here) am not sold on using a template engine. It gets the job done on initial render, but then you have to fill in the "update" cracks with all this imperative jquery code to sync state to DOM state. Data changes all the time (user input, server events, fetching data, fetching components, etc.), this becomes a nightmare very quickly with HTML templates.

Yup.

Widget as UI Framework

When it comes down to it, I found that widget needs to either not do anything (allow import, then let the use decide how they want to render the DOM) or do [almost] everything. Doing everything would include providing a framework to abstract the DOM and providing primitives to represent a UI and how it connects with data in the database. Perhaps this would look more like Vue templates where DOM attributes have significance outside of HTML.

Whoa I hadn't seen Vue but it is pretty awesome. Reminds me of knockout.js from back in the day, but .. better. I really dislike React's "lets just make everything Javascript" paradigm. But Vue seems like a really nice balance.

<li sync-rows="endpoint.session">{{row.id}}</li>

just conceptual.

I'll refer to these two options as "Framework that does nothing" and "Framework that does everything".

Server-Side Rendering

Framework that does nothing

I didn't start here because there is probably no clean solution. SSR with widget code that supports modules needs to run untrusted code in Node. Period. This is fine if you only run code from a trusted source, but is a horrible security flaw that could never be overcome otherwise. So far, we haven't proven the need to run OPC (other people's code), so I still like the possibilities with this idea.

Framework that does everything

A framework that abstracts DOM from the user could construct the HTML document quite simply. The only JS the user would write are event handles that can be bound on the client.

Data fetching

right now there is no way to do pure server-side rendering that uses server-side database calls

This was a big reason for me to rewrite datum to run in both the browser and node. I think whatever widget looks like, the datum layer needs to run in all environments.

With both of the options I outlined, data fetching in SSR is possible. This requires using the same datum layer on the client as on the server.

Agreed. I took a crack at rewriting datum for plv8, with the same API but different backend calls under the hood. Got most of the way through it but it needs a little work.

Conclusion

This is less structured than I wanted it to be. Hopefully I made a little sense.

Some. :-)

I think the main focus should be to identify what the ideal widget solution looks like.

Great list of questions. First thoughts:

Does "untrusted" code run in node/plv8?

I could imagine some future scenario where a plv8u language exists. Or a pl/quickjsu for that matter. But I think having a very secure v8 sandbox is a very nice compliment to the very secure browser sandbox, and the very secure postgresql user model. For doing insecure stuff, there is always the pl/pythonu language, and Python imho is wildly more mature IMHO.

What level of DOM manipulation do users need to do?

Not sure I follow this one. Developers need full control of the DOM, right? Am I missing something? Are you thinking of more like a wix-style widget where they're dragging and dropping and such? More WYSIWYG type stuff? That would be awesome but if we play our cards right, we can build that on top of a nice widget system.

How do users use code from the community? Does the user bring their own tools (vdom rendering, etc.) or is everything provided for them? Can users do both? Ooh... power users can use a library, end-users can use a visual programming interface?

Yeah, still very much focused on power users at this stage, and they can use whatever JS modules they desire, as long as they can run in plv8. I think?

From what I've already built, I think server-side data fetching could fall into place for most solutions. The only issue would be plv8 versus node. I'm definitely weary of trying to make plv8 somehow work with modules. To be honest, I don't see the value add for putting framework code in a PL lang.

Well, to me the v8 sandbox makes it compelling since running other people's code, and a big distributed network of code sharing, is a goal...

Modern JS frameworks (and honestly, something like ClojureScript) are really good examples of ways to solve the UI space.

Will check out ClojureScript.

Good stuff, thanks for the brain dump. Above are just some first thoughts. I'd be down to just pitch widget.js entirely if we can find something that does a better job. Where I'm concerned is... well, so there are reasons widget was designed the way it was that are more about the bigger picture goals of the project, and not so much about building a better node, or anything like that. Meeting the bigger goals of the project -- coupling schema w/ components via semantics, doing version control on widgets, reusing widgets from other bundles in my project, stuff like this -- that I want to make sure we retain. Which means I should probably articulate what those are again.

In your conception of widgets, would everything still be able to be stored in the database, versioned, passed inputs, etc? Also, import maps are really powerful, and maybe they could be leveraged so that we build one map for the browser environment and one map for the server-side environment, so that both environments use the same set of modules in the database, and code can be written so that an import statement is the same in either environment, but mapped to resolve to the appropriate resolution mechanism be in server-side or in the browser.

Half-cocked braindump. Gotta run. Thanks.