RFC: SSR tools

[PaulMaly]

Hi guys!

I want to suggest a few additional methods for the Ractive's core which are very useful in SSR and isomorphic (universal) approach.

Wait:

/**
 * Add asynchronous value to waitings.
 *
 * @param {promise} value - Asynchronous value
 * @param {string} key - Optional key name in data object
 */
Ractive.prototype.wait = function(value, key) {};

Ready:

/**
 * This callback fired when all waitings resolved.
 * @callback readyCallback
 * @param {object} err
 * @param {object} data
 */

/**
 * Ready event fired when all waitings resolved.
 *
 * @param {readyCallback} callback - The callback that handles the event. (node style)
 * @returns {promise}
 */
Ractive.prototype.ready = function(callback) {};

How it works:

For example, we've multi-level components hierarchy and some of these components fetches data from the server (or make another async operations):

const User = Ractive.extend({
    data: () => ({
         user: null
    }),
    oninit: function() {
         fetch('/users/me').then(res => res.json()).then(user => this.set('user', user));
    }
});
const Products = Ractive.extend({
    data: () => ({
         products: []
    }),
    oninit: function() {
         fetch('/products').then(res => res.json()).then(products => this.set('products', products));
    }
});

const app = new Ractive({
     el: '#app',
     template: `<User /><Products />`,
     components: {
          User,
          Products
     }
});

On the client, we don't have any problems with this approach, because the client is fully asynchronous itself. Servers-side (nodejs) is also asynchronous, but to do SSR we need to make a decision when we need to halts and call toHTML() method.

if (NODE) {
     const html = app.toHTML();
     ....
}

In this example, if we call nodejs-specific code right after app creating we'll get empty html string because asynchronous operations most likely weren't executed yet. Also, we can't just use hacks like setTimeout() because it will only generate race conditions.

There are two competing solutions of this issue:

First, we can take out asynchronous operations from the components and make prefetch. Like we do it on PHP or other synchronous platforms. But to be isomorphic (universal) we also need to do this on client-side too. Which is not so good idea, I think, and also it's not the right JS-way.

The second way I suggest. A full code could look like:

const User = Ractive.extend({
    data: () => ({
         user: null
    }),
    oninit: function() {
         const user = fetch('/users/me').then(res => res.json()).then(user => this.set('user', user));

         // now we just say - wait for this operation, please.
         this.wait(user); 
    }
});
const Products = Ractive.extend({
    data: () => ({
         products: []
    }),
    oninit: function() {
         const products = fetch('/products').then(res => res.json()).then(products => this.set('products', products));

          // if we doesn't need to wait some operation to do SSR - just don't add it to waitings.
          // this.wait(products);
    }
});

const app = new Ractive({
     el: '#app',
     template: `<User /><Products />`,
     components: {
          User,
          Products
     }
});

if (NODE) {
     // pass callback to handle when all waitings would be resolved and get html
     app.ready(function(err, data) {
            const html = this.toHTML();
            ....
     });
     // or use promise-style
     app.ready().then(...).catch(...); 
}

This simple approach helps us to resolve not only this SSR issue, but it also resolves one more - double data fetching. When our js code init on the client it needs to decide whether there is a need to fetch data or data already was fetched on server-side. When our app asynchronous and has multi-level components hierarchy it's could be not so simple problem to solve. But we already solved this problem with these two methods, check it out:

// just pass the second parameter to wait() method to define the key name of this data
...
this.wait(user, 'user'); 
...
this.wait(products, 'products');
...
// after that, all these data will be available in structured look as the second parameter in the ready-callback:

app.ready(function(err, data) {
     console.log(data); // { User: { user: {} },  Products: { products: [] } }
     const html = this.toHTML();
     // somehow put this data to the result html, for example in window.__DATA__ object
});

It also awesome works with new {{#await /}} blocks, so full code could look like:

const User = Ractive.extend({
    template: `
         {{#await user}} Loading user...
         {{then data}} {{ data.username }}
         {{/await}}
    `,
    data: () => ({
         user: null
    }),
    oninit: function() {
         const user = this.get('@global.__DATA__.User.user') || 
                      fetch('/users/me').then(res => res.json());
         this.wait(user, 'user'); 
         this.set('user', user);
    }
});
const Products = Ractive.extend({
    template: `
         {{#await products}} Loading products...
         {{then data}}
                {{#each data}} {{ .title }} - {{ .price }} {{/each}}
         {{/await}}
    `,
    data: () => ({
         products: []
    }),
    oninit: function() {
         const products = this.get('@global.__DATA__.Products.products') || 
                          fetch('/products').then(res => res.json());
         this.wait(products, 'products');
         this.set('products', products)
    }
});

const app = new Ractive({
     el: '#app',
     template: `<User /><Products />`,
     components: {
          User,
          Products
     }
});

if (NODE) {
     app.ready(function(err, data) {
            const html = this.toHTML();
            // somehow put this data to the result html, for example in window.__DATA__ object
            ....
     });
}

If these suggestions will be accepted, I can provide the pull request for these features. Actually, I already have that code as a separate module.

Expect your comments, thanks!

[fskreuz]

Suggesting rewording the post to begin with example of the proposed API first, then explain afterwards. I got lost in the middle reading everything. I like the idea, but here's a few things:

First, we can take out asynchronous operations from the components and make prefetch. Like we do it on PHP or other synchronous platforms. But to be isomorphic (universal) we also need to do this on client-side too. Which is not so good idea, I think, and also it's not the right JS-way.

Your concept of SSR is funny. As far as I know, SSR runs like this:

1. On the server, givenstate, the lib will render the components into HTML.
2. On the client, given state, the lib will initialize the components onto already-rendered HTML.
3. For the client to reuse as much of the already-rendered HTML as possible, it must be initialized with the same state in order to get the same tree.
   • For nodes it can't reuse/tree differs, they're discarded and new ones are created.
4. For that same state to get to the client side, it must be serialized and embedded on the page.
5. For that same state to be serialized, the data must have been fetched in full.

And so, on the server-side, wrapper code would look like this:

app.get('/some/route', (req, res) => {
  // some logic

  getData().then(data => {
    const app = App({ data })
    const html = app.toHTML()
    const state = JSON.stringify(data)

    res.someFnThatRendersHtmlAndEmbedsStateInPage(html, state)
  })
})

On the client-side, the bootstrapping code would look like this:

document.addEventListener("DOMContentLoaded", function(event) {
  const state = document.getElementById('hidden-input-with-stringified-state')
  const data = JSON.parse(state)
  const app = App({ el: '#container-with-rendered-html', data })    
});

And all the components would remain isomorphic/universal. They all assume data is already there fetched by some wrapper code, not the components themselves. Any async data that you want to be pre-rendered on the server and reused on the client should be fetched in advance and serialized into state that the client can use on initialization.

Anything dynamic/async-on-init would just not render on the server, and will only be rendered on the client. It doesn't make sense rendering something on the server that will get trashed on the client because it didn't have that data during initialization.

[MartinKolarik]

Your concept of SSR is funny. As far as I know, SSR runs like this:

That's really just one way to do it, exactly as @PaulMaly said. I usually use this way, but the alternative does make sense sometimes - you can fetch the data directly in components, which means you don't need two different wrappers (aka controllers) for server and for client to get the same data (important point here is that in a SPA you don't only have the initial state, but also need to fetch more data based on user actions).

That said, I don't think this is something that belongs to core. It's something you can easily add yourself, and if you need it in more projects, you can publish it as a module (I myself have a few of such extensions, which I will probably cleanup and publish later after testing them in some internal projects).

[PaulMaly]

@fskreuz

Your concept of SSR is funny.

Why it's funny? Could you please explain it?

As far as I know, SSR runs like this: On the server, givenstate, the lib will render the components into HTML. On the client, given state, the lib will initialize the components onto already-rendered HTML. On the client, for the lib to reuse as much of the already-rendered HTML as possible, it must be initialized with the same state. For that same state to get to the client side, it must be serialized and embedded on the page. For that same state to be serialized, the data must have been fetched in full.

This is "the first approach", which I also mentioned. Also, I explained shortly why I believe that this approach is not the best for web apps. I can explain more deeply if you need it.

Anyway, I don't suppose that the components must definitely look like just a wrappers around html snippets. As I said in an initial comment - my components doing async operations. Not just a receiving the state from the outside. How do you plan to use your approach in the case, I described?

For example, we've multi-level components hierarchy and some of these components fetches data from the server (or make another async operations):

Further, your code already not enough isomorphic, because you use server-side routing directly. Instead of using isomorphic routing inside the shared code. If you won't do that, you will face with the problem that you can't just getData(). ))))

 const app = App({ el: '#container-with-rendered-html', data })  

Just to be clear, could you please explain how exactly you plan to initialize all the tree of your components with this data?

@MartinKolarik yep, it seems you've understood the idea correctly.

That said, I don't think this is something that belongs to core. It's something you can easily add yourself, and if you need it in more projects, you can publish it as a module (I myself have a few of such extensions, which I will probably cleanup and publish later after testing them in some internal projects).

Just like I did, but I've thought that 2 additional methods are a quite reasonable price considering opportunities it gives. So I've decided to share it with the others. To be clear - implementation adds about 50 line of code and mostly use already built-in functionality.

p/s In my isomorphic apps, I've about 95% of shared code from absolutely whole the code of an app (including all server-side specific code). Moreover, in my code, you won't meet any silly checks like if (IS_SERVER) / if (IS_CLIENT). The code is very clean, because of approach I use.

[MartinKolarik]

Just like I did, but I've thought that 2 additional methods are a quite reasonable price considering what it gives opportunities. To be clear - implementation adds about 50 line of code and mostly use already built-in functionality.

I don't think it would be useful for most people using Ractive, and again, if it can be implemented as a plugin, I see no point. Tbh, in the feature I could easily see the whole .toHTML() implementation being moved out to a separate tool. I think there's rarely a reason to use it on the client, and it would probably be rather easy to implement as a separate tool.

[PaulMaly]

@MartinKolarik

Tbh, in the feature I could easily see the whole .toHTML() implementation being moved out to a separate tool.

It would be really strange move, if we'll remove SSR functionality, while all the others implements additional features in this direction.

I don't think it would be useful for most people using Ractive

No doubt, it would be useful for those people who doing isomorphic apps or even just SSR. Ractive already has many features which I never been used and seems won't. But I can imagine that they are useful to other people.

[MartinKolarik]

Ractive already has many features which I never been used and seems won't be. But I can imagine that they are useful to other people.

And what I'm saying is I think we should be trying to move such features out to plugins, not in.

while all the others implement additional features in this direction.

It was really just an idea, but removing it from core might be the oportunity to do just that,

[evs-chris]

I think moving SSR into a separate module within core would make a lot of sense. We've been trying to come up with a good way to modularize for a while now, and toHTML is a pretty good candidate to try it out. I think messaging (moving to error/warning codes and having the actual strings in a separate module) would also be a good target because Ractive is real chatty with warnings. Add that to the parser that's already separated out and we could have a build matrix to pick minified and what modules over core to include. The default ractive/ractive.js build would still be the kitchen sink build, but it would allow a bit of optimization for client side where you probably don't need SSR, parsing, or the ability to get full warning and error messages once you hit production.

[PaulMaly]

@MartinKolarik

And what I'm saying is I think we should be trying to move such features out to plugins, not in.

I'm not sure, that most of these features could be implemented as plugins. For example, I never used Anchors and attachChild/detachChild because I think that declarative code is the best way to keep it clean and simple.

I'm not sure that there's the case where I need to use such things like findParent()/findContainer(). All these stuff is too "imperative" to me and looks like excessive manual control.

But SSR on hype right now and I think Ractive needs to provide built-in solutions for the main issues of this approach. And these two methods do a lot of dirty work, at the small additional code size.

[PaulMaly]

@evs-chris I like the approach which was introduced by @fskreuz in some issue (I can't find). It was about code splitting to separated modules inside the core to be able to import only such functionality you need. This is the right way, but it affairs of next days.

Right now we have mostly monolith, so I suggest just add two small but very-very useful methods to it. If you'll decide to split this monolith to separated sub-modules in future, these 2 methods won't prevent you to do that.

[fskreuz]

I think that declarative code is the best way to keep it clean and simple.

Look what happened to Grunt and soon, webpack 😁 Just had to deal with Grunt earlier today. Not the best thing ever.

Why it's funny? Could you please explain it?

Correct me if I'm wrong, but if I remember correctly, when Ractive enhances HTML, it tries to match up the tree in the HTML and the tree it's trying to generate. If it sees something's off, it will throw away the subtree from the HTML and recreate that tree as if no enhance was done for that subtree. If the HTML was rendered with state that caused a subtree to render on the server, but Ractive was initialized with state that makes that subtree disappear on initialization on the client, it throws away that HTML subtree, effectively removing the benefit of SSR and HTML reuse.

Just to be clear, could you please explain how exactly you plan to initialize all the tree of your components with this data?

Single state object from server fed into a Redux store/redux-like storage. All components either subscribe to the store or gets passed down data that's also from the store. Once everything is set, instance is created with all that data. Works wonders for your skin. 😁

---

All funny things aside (I blame this morning's coffee, or the lack of it earlier):

1. I think SSR should be its own module. I think "add-on" is the proper term instead of plugin. a. Base Ractive, skinny, like Svelte/Inferno. All data-ops/AST mangling, no DOM, node, nothing. b. Add on SSR, or DOM, or something. c. ??? d. Profit!
2. Which needs to a modular Ractive, like how RxJS is broken down.
3. Which needs to a rewrite, proposing TypeScript for early safety and strict syntax/structure.
4. Which needs to a monorepo, because we'd like to test all these at once.
5. Still have a kitchen sink build for all the things basic/just-add-water demos.

Everything's pretty much in place really 😁 Just need to dump in more time and coordination.

[fskreuz]

Disclaimer: Not a regular SSR person. I use Ractive on different backends (Spring, PHP, Node, Drupal, name it, we probably have it). SSR is probably the least of my concerns. I just need a library that can be dumped into any setup (Grunt, Gulp, Webpack, none) and Just Works™.

Angular 1.x pretty much fits the bill except that it's overkill in most cases, has a relatively steeper curve than Ractive, and lots of conflicting info everywhere (use this, no don't use this).

[PaulMaly]

@fskreuz

Look what happened to Grunt and soon, webpack 😁 Just had to deal with Grunt earlier today. Not the best thing ever.

Hm, and what happened with Webpack? I use Webpack in my projects and he's very vigorous. I'm not sure that there's a better tool for building/packagins of web apps.

Correct me if I'm wrong, but if I remember correctly...

And? Why do you think that it's funny? Also why do you think it's not working in my concept?

Single state object from server fed into a Redux store/redux-like storage. All components either subscribe to the store or gets passed down data that's also from the store. Once everything is set, an instance is created with all that data. Works wonders for your skin.

First of all, not all developers use Flux-architecture, and things like Redux/etc. I one of them and I never got necessary to use global store in Ractive. I think it's compelled approach when you deal with such primitive library as React. But Ractive is powerful enough to handle it by itself.

Second one, could you please explain how using Redux essentially differs from window.__DATA__, in context of SSR/isomorphic? Ok, all the components subscribe to the store and all that stuff, but why they just can't be initialized from window.__DATA__? Maybe you just don't clearly understand:

// you do
getData().then(data => {
    const app = App({ data })
    const html = app.toHTML()
    const state = JSON.stringify(data)

    res.someFnThatRendersHtmlAndEmbedsStateInPage(html, state)
});

// I do
app.ready().then(data => {
    const html = app.toHTML();
    const state = JSON.stringify(data);
    res.someFnThatRendersHtmlAndEmbedsStateInPage(html, state);
});

Almost the same, but your getData() is abstruction and definitely has much more reefs you could imagine now. My ready() is always the same. Further:

// you do
const state = document.getElementById('hidden-input-with-stringified-state')
const data = JSON.parse(state)

// I do
const data = window.__DATA__;

And the last one, I asked you specifically about this line: const app = App({ el: '#container-with-rendered-html', data }). Seems that App is the Ractive's instance. So, I don't know any good automatic way to propagate root instance data to all that child components. So if you just suppose to use Redux inside of that instance, it's fine. But then I really don't understand what the difference?

Except that your approach supposes to use only "prefetch"-way of doing async things. That forces you to have many code duplications on server-side and client-sides.

Instead, my "funny" approach supports any type of async operations: you able to prefetch some data, you able to bury async things far in a hierarchy of components. It doesn't matter. All that you need to do is use wait() method to get all that data in one place and ready() callback to SSR your app.

All funny things aside (I blame this morning's coffee, or the lack of it earlier):

Yes, I agree almost with all points, but how it's corresponding to the current situation and my suggestions? I don't even ask you or @evs-chris to write this code, it's already written. I suggest to implement it to Ractive now. And I believe, if sometimes your great ideas about code splitting will be embodied, these lines of a code won't prevent you to do that.

Disclaimer: Not a regular SSR person. I use Ractive on different backends (Spring, PHP, Node, Drupal, name it, we probably have it). SSR is probably the least of my concerns. I just need a library that can be dumped into any setup (Grunt, Gulp, Webpack, none) and Just Works™.

Well, isomorphic approach means the existence of at least 2 servers: front-end server on NodeJS, and your favorite back-end server (name it). So, seems, you preach absolutely a different approach, therefore, I think it's not so close to you.

[dagnelies]

I also don't fully get this one. As far as SSR goes, I wonder why use a client lib for it instead of plain simple templating. But that's another topic.

Besides of this, I think your examples introduce a lot of complexity because you fetch data within your components. IMHO that's the best way to make your components not reusable, to mix logic with templating and to make it harder to debug. Instead of this, it would be way easier to fetch data outside and render it once you got what you need, especially for SSR.

For client side rendering, just fetch it async and the content will be displayed when it's there, and you can display placeholders in the meantime using a dumb {{#if}}...{{else}}.

[PaulMaly]

@dagnelies

I also don't fully get this one. As far as SSR goes, I wonder why use a client lib for it instead of plain simple templating. But that's another topic.

This talk is not only about SSR itself, but more about an isomorphic approach which means using exactly the same code of the app on both sides - client and server. So, a key indicator of the good isomorphic app - quantity of the shared code between client and server sides. My current stable result is about 95% of the shared code. Other 5% includes server-side specific code (generally common for all my projects) and DOM related things. I believe it's an excellent result.

Besides of this, I think your examples introduce a lot of complexity because you fetch data within your components.

Hm, I'll try to explain. Basically, there're three types of the components:

1. Pure components - simple components which depends only on input params (attributes/props/etc.) and contains only view level logic. That's what you talking about, but...
2. Autonomic components - complex components which implement some feature in "low coupling and single responsibility" way. Often, they also use pure components inside of themselves and contains specific top-level logic, fetches data, etc.
3. Wrapper components - not isolated components which are most often used to improve templates structure, passing params down through, etc.

Any type of components doesn't exclude others and all them should be applied in the right way and in place. A good example, I guess, is <Articles/> component which implements a list of the articles with filters, pagination, etc. and <Tags/> component which implements some list of the tags.

Key difference between them (1) complexity of components, (2) <Articles/> component has always the single data source (eg. api endpont /articles) which can be used with different params (eg. filters/page/etc.). There's no any sense to take out data-fetching outside of this component, it will only complicate interactions. A top-level component has to manage this component only using params (props/attributes). Not just to pass prepared data down.

Contrary to that, <Tags/>component able to has many data sources, eg. common tags-list or specific article tags, etc. So, we definitely need to take data-fetching out of him. Good that, tags basically are just an array of keywords and in most of the cases doesn't need to fetch them in a separate way. They can be a part of an article data or something.

IMHO that's the best way to make your components not reusable, to mix logic with templating and to make it harder to debug.

Why do you think so? Btw, it's even simpler, than reuse of pure components. Check this out:

// to simple reuse <articles/> I need just:
Ractive.components.articles = require('./components/article');

// and paste the html tag in my template in any place I need, that's it
<articles params="{{ params }}" />

// to reuse <tags/> I need to previously fetch the data in each place I use it:
Ractive.components.tags = require('./components/tags');

// first place
fetch('./tags').then(res => res.json()).then(tags => this.set({ tags }));

{{#if tags}}
<tags tags="{{ tags }}" />
{{/if}}

// second place
fetch('./article/1').then(res => res.json()).then(article => this.set({ article }));

{{#if article.tags}}
<tags tags="{{ article.tags }}" />
{{/if}}

I don't think that <Tags/> component easier to reuse than <Articles/> component in this case.

Instead of this, it would be way easier to fetch data outside and render it once you got what you need, especially for SSR.

Especially for SSR, your proposal the best way to make your code less isomorphic and the statement below only confirms it:

For client side rendering, just fetch it async and the content will be displayed when it's there, and you can display placeholders in the meantime using a dumb {{#if}}...{{else}}.

So, seems you suggest to prefetch the data for SSR and fetch data in an async-manner on the client? But for what purpose if it could be the same code? And why you think this approach would be way easier? Let's look, for example, you already have client-side async data fetching, alright?

// you're doing that somewhere you want

fetchArticles().then(articles => this.set({ articles }));
...
// other place
fetchTags().then(tags => this.set({ tags }));
...
// one more
fetchUser().then(user => this.set({ user }));

ok? So, to reuse this code on server-side in my approach you just need to update it like this:

// you're doing that somewhere you want

this.wait(fetchArticles().then(articles => this.set({ articles })));
...
// other place
this.wait(fetchTags().then(tags => this.set({ tags })));
...
// one more
this.wait(fetchUser().then(user => this.set({ user })));

It's not so complex aren't you? After that you'll be able to do SSR in ready() callback:

// somewhere in server-side specific code
...
r.ready(function(err, data) {
     const html = this.toHTML();
    const styles = this.toCSS();

      ....
      res.send();
});
...

You think it's more complex than write two separate data-fetching strategies, really? Sorry, but I don't think so.

[PaulMaly]

I’ll publish SSR tools via Plugins soon.