Nested Templates (and wrapping JavaScript with templates)

[jbolda]

I started some work trying to get templates to "wrap" pages that are written in JavaScript. This is useful for those writing their blog posts in JavaScript (or any page with repeatable and similar elements). As it stands currently, the structure is effectively html.js(layout(JavaScript Post)). Any repeatable elements need to be included in each JavaScript file. The ideal structure would be html.js(layout(template (JavaScript Post))).

The current structure makes sense for markdown and other formats as a structured format can be "mapped" into a template file: html.js(layout(template with data mapped into it)). The template can map the data into a typical structure, and multiple templates can even be composed together via typical React methods.

As JavaScript posts replace the template, we lose some of that composibility compared to other formats. I started modifying the code to use both a template and the JavaScript post to solve this. More and more it was feeling forced though. I was adding checks to see if it was a JavaScript file, but then it does not give you the option to opt out as easily. createPage() really only accepts one component as well so there wasn't a great way to specify both a template and the JavaScript file. Again, this makes sense in markdown as we are using graphql to pull in the data. It feels out of place to query for a React component though.

We really just need a good way to require the component directly. And well, we have that already with templates and layouts. So why can't we just use this? If we had nested templates, we could deal with this situation by specifying the template and then specifying the JavaScript post. In terms of implementation, the JavaScript posts can act much like they are a template without any graphql. Now this seems to fit very well into our mental model.

And so began adding nesting templates. I have started a branch that I can eventually PR. No promises on the speed until I'm finished. We can add it as a branch on the main gatsby repo if anyone wants to help. I have updated using-javascript-transforms to prepare for the nested templates and testing it. I started using an array that the createPage() component can accept to specify the nested templates. I've just been fixing the errors that come up until it runs. I've gotten through the redux stuff so far.

Does anyone have alternative thoughts on how to implement any of this? Feedback appreciated. Thanks everyone.

[jbolda]

From @KyleAMathews in #1895 (thought it made sense to discuss here)

Hey just had a quick look at this and wanted to correct a miscommunication. We want nested layout components not nested page components. It's perhaps more semantics than anything else but there can only be one page component controlling the page at a time. The boundary that's significant is between pages and layouts as layouts stay constant across page changes which lets you retain state in the layout as you click between pages. Having nested page components doesn't gain you anything because you can already nest components quite easily yourself.

In the case I described above, it really ends up not being semantics. We had discussed in discord that the layouts are intended to be site wide and mostly non-page specific, and templates are to focus on page specific data. I think I might clarify a bit... I believe it's most correct to say layouts tend to stay the same across route changes, but templates will change with the route as page data is tied to that route.

My understanding was that both nested layouts and nested templates are both potentially useful and an eventuality. To be honest, I'm not sure what additional value we get out of nested templates over and above the default react methods where"you can already nest components quite easily yourself." With @craig-mulligan PR for layouts, you should really only need like one level deep of nesting for site wide data especially when you can specify which layout you want to use (which actually breaks the paradigm that layouts are not dependent on the route).

Nested templates deliver a lot of value if you are mixing your data types. Using-javascript-transforms provides a good example of this (and it is now updated for this PR). Templates seem to fill two roles: either converting non-react data (markdown, etc.) into a react component via props AND/OR putting those (possibility newly created or existing) react components into a structure. If you are using one single data type, such as markdown, one template can mostly fill both roles. You may need to have multiple templates with duplicate information or pass between components via the react way. It starts to get more ugly if you need multiple queries and you pass lots of data through the components. BUT still doable.

It is pretty much a hard stop with JavaScript pages though. The page IS the template in the current Gatsby. This means that any sidebars or footers, etc. that makeup the structure of the page and require data from a query at that specific path will need be included in every JavaScript page. For something that previously supported it in v0, and touts the reactjs foundation, it feels odd that you can't split these concerns up. If I want to move an "author" component from the bottom to top of the page, I would need to update each post separately.

Coming back to your mention of state. Each route still has one piece of data. The state is the same for that route. It is just segregated by additional components keys, and fed into each component separately by that key. Passing data around in react is not bad, but it clearly becomes cumbersome at some point otherwise redux, et al. would not have been born. It seems to make sense to me, just from a DX point of view, to take advantage of the gatsby graphql niceties and let that manage our "state" a little more. Or at least give the option. I don't see any performance reasons for either case, but componentizing and splitting up your concerns into small pieces seems very reactjs to me.

Long story short, it was a deliberate choice to work on nested templates. After it was clear that layouts would not solve any of the aforementioned issues, as discussed here and in discord, I understood that the consensus was both nested layouts and nested templates should be added. My apologies if our understanding was misaligned. Nested templates are so much more valuable to me (and others have said the same), as composing page "state" gets much more involved then sitewide layouts.

[sebastienfi]

I will start by saying thank you for this deep and clear explanation of the use-case you intend to solve.

Could you please precise what would be the reason to use JS for Posts instead of Markdown or any other source that has a Gatsby Source plugin ?

I don't pretend to have narrowed your use case, because this seems quite complex and surely I understood only what I was able to understand, but it seems to me that you should move to another kind of Gatsby Source if you intend to use a lot of JS files inside the Page folder. As I see you consider the below Scenari C as the only way to do DRY code while using a large amount of pages, but it seems to me that this is already achieved using other sources maybe more adapted to handling a lot of Posts. Gatbsy Sources plugins can also be mixed. Templates are just what you need.

I think I can identify one reason why we would prefer Nested Layouts over Nested Templates. I believe we all mix the content source (data types), because we all agree to say that the home page is somehow always different from the rest of the pages. I believe it to be the reason for the two following original scenarios :

Scenari A
html.js
  |  Layout / index.js
    |  Page / index.js and {any-other-js-page}.js

Scenari B
html.js
  |  Layout / index.js
    |  Template / index.js and {any-other-js-page}.js

Scenari A targets the home page and Scenari B targets the other kinds of contents. Sometimes Page / index.js is a basic copy-paste of Template / index.js.

Is this scenari below what you want ?

Scenari C
html.js
  |  Layout / index.js
    |  Template / index.js and {any-other-template}.js
      |  Page / index.js and {any-other-js-page}.js

Why not doing this ?

Scenari C
html.js
  |  Layout / index.js
    |  Page / index.js and {any-other-js-page}.js
      |  Any React Component, shared accross several pages and hosted in, let's say, `src/components`

This way you pass data to your React Components as Props and you end up with a "Dumb Component" hydrated with data from above ? You could also use a unique Props, and use PropTypes.Shape to describe the data inside ? I know this is not ideal because you still have to repeat the calls to your components, but you could make a kind of HOC Component called in the Pages, wrapping your data, and taking props...

[jbolda]

I believe you mostly understand the situation. I think in general it feels out of place to say "use markdown" when one of the selling points is react. Ignoring this, I think there are some technical benefits to be had. I'll touch on those on a moment, but first... I don't think I'm clear regarding your example of nested layouts. Both scenario A and B are possible in the current version of Gatsby. I am already doing your second version of scenario C. It is pretty poor dev experience though. I need to include the dumb components on every post, and include a graphql query at the bottom of every post. If my dumb component changes and expects more/different data, then I need to update the query on every page. Each post becomes dependent on the structure and data surrounding it. This feels like an anti-pattern. The content should be able to be pulled out and dropped in somewhere else. This is one of the great benefits of markdown for instance. If I need to bring dumb components and graphql to every post, that is the definition of template. Nested Templates is the solution to fit it into the Gatsby definition of templates.

Anyways, let's see if I can represent the possible technical benefits and a real world scenario for it. It is very clear that Gatsby is great for portfolio websites as many newcomers will use it as a first test. With the rise in popularity of visualization libraries, browser games, AR, and VR, all in the browser, Gatsby can knock it out of the park there.

With these applications, performance and control are valuable to make a good experience. A solid example is having multiple visualizations on one page, all dependent on one set of data. If I'm writing this in react, I can fetch the data, and then each visualization can use that data. Very simple and how you would normally do it in "vanilla" JavaScript.

Now, if we encouraged use of markdown and we added support for react components in markdown (I don't think we have figured that out yet?), we could get each react based visualization to render, but it would need to fetch the data for every visualization. Well, what if we just put the data in graphql? But then we need to fetch it in the template. If I'm doing 100 tech demos, maybe we can reuse some of the templates, but we might have to fit into 5, 10 , 20 different templates? If this person is an expert in A-Frame putting it on a JavaScript page, why make them jump through a markdown hoop?

Additionally, I think there is performance concerns as well in current Gatsby using your second version of scenario C. I believe if react rerenders something in the DOM, it doesn't actually diff the children. I'm not certain about React 16, but I'd have to research to confirm. Please amend this if I'm incorrect. That being said, in your second version of scenario C, when you switch between routes, the whole page including your dumb components will rerender. With nested templates, the (previously dumb) components sit above your page. When the route changes, the "template" is the same so that doesn't change, it passes the new props, and then updates just the data. Again, I'll have to do more research on that if need be.

Another consideration: a Gatsby ideal is to make amazing performance the default. We have greater control over improving performance for layouts and templates. If we start preaching react components in markdown, I fear that paradigm of composition will lead us away from being able to achieve that ideal.

[jbolda]

[RE: second to last paragraph above] React will destroy the children if the node type changes, otherwise it will replace the data: https://facebook.github.io/react/docs/reconciliation.html#the-diffing-algorithm. I can't find anything about react16 though. From what I read, it will do much the same though.

That being said, switching between a route with a root data type of markdown to a route with a root data type of javascript will likely rerender that whole component. Not a big deal when switching between say the homepage (src/pages/index.js) and a blog post (src/articles/my-first-post/index.md), as most of the template type elements likely need to be rerendered. However switching between blog posts with different data type roots (src/articles/my-first-post/index.md VS src/articles/my-second-post/index.js), will likely rerender more elements than necessary.

Presuming I am understanding it all correctly, nested templates do have a potential performance increase.

[KyleAMathews]

BTW, the entire React tree gets rendered on every page change including the layout component. The distinction between pages and layouts is layout components aren't unmounted on page changes (unless of course you're changing layouts) whereas pages always are.

I again don't see the necessity of supporting nested page components as you can already easily share components across pages. Nor, after thinking about it more, see any reason for nested layouts as again, sharing components across layouts is easy + it's easy to change the layout based on the current pathname or pass data from the page to the layout.

I'd love to create an example site for this but in the meantime this should help https://github.com/gatsbyjs/gatsby/issues/2112

[jbolda]

To understand the expectations of composing things, I made a couple diagrams. Just going to ignore the perf discussion for now.

This is, as far as I know, the current best way to set things up if your blog posts are made of markdown and javascript as the root data type. With the markdown, you will use componentA and componentB once each, and with javascript, you will use them each N times, where N is the number of blog posts.

diagram 1

                            htmljs
                              +
                              |
                              +
                     src/layouts/index.js
                              +
                       +------+------+
                       +             +
 src/templates/mdBlogPost.js      src/articles/my-first-post/index.js
+--------------------------+      +---------------------------------+
|                          |      |                                 |
|     <div >               |      |     <div >                      |
|      <componentA/>       |      |      <componentA/>              |
|      < markdown data />  |      |      < whatever other jsx >     |
|      <componentB/>       |      |      <componentB/>              |
|     </div >              |      |     <div />                     |
|                          |      |                                 |
+--------------------------+      +---------------------------------+
  where this component pulls in      where every post needs to have the
   any markdown data and is           componentA and componentB as part
   reused for every post              of it

Is there a better way then this? This does not seem easy to manage to me. The only alternative is to use a HoC within in javascript post such as the following:

diagram 2

<HoC {...pagedata}>
  < whatever other jsx >
</HoC>

where HoC is:

<div>
  <componentA/>
  {this.props.children}
  <componentB/>
</div>

You still need to include this component and a query on every post that you create.

With the nested templates, it looks like the following. It seems much nicer to compose things in my opinion.

diagram 3

                            htmljs
                              +
                              |
                              +
                     src/layouts/index.js
                              +
                              |
                              +
                  src/templates/blogPost.js
               +-----------------------------+
               |                             |
               |    <div >                   |
               |     <componentA/>           |
               |     {this.props.children()} |
               |     <componentB/>           |
               |    </div >                  |
               |                             |
               +--------------+--------------+
                              |
                              |
                       +------+------+
                       +             +
 src/templates/markdown.js        src/articles/my-first-post/index.js
+--------------------------+      +---------------------------------+
|                          |      |                                 |
|     <div >               |      |     <div >                      |
|      < markdown data />  |      |      < whatever other jsx >     |
|     </div >              |      |     <div />                     |
|                          |      |                                 |
+--------------------------+      +---------------------------------+

Is there an alternative to the diagrams shown here? I want to confirm that the intention is to tell users to use diagram 1 and/or 2. My understanding is that the con for diagram 3 is that we have diagram 1 and 2, and we don't want to try improve upon that. Is this correct?

[jbolda]

I forgot to mention... componentA and componentB are for route specific data. componentA might be for an image, dates, and what category it falls under (looking to category pages). componentB might include author information, which post to read next, and comments about the current post.

And I am subscribed and have been following #2112. It can make sense in some cases where say, you want to tell the nav bar to highlight an item as you are currently in that page. I thought it was decided though that layouts are site wide and templates are route specific. It doesn't make sense to me to throw componentA and componentB on the layout and try to pass props up to it... but only when you are using JavaScript.

[jbolda]

I spoke some with @sebastienfi to better understand his use case (thanks again!).

@KyleAMathews, it sounds like you just don't want to support this use case as HOC and graphql fragments are "good enough." This presumably won't benefit enough users to add to the core. If you could confirm, I will just close #1895.

[KyleAMathews]

Good enough + we have to tread carefully when adding Gatsby-specific architectural things as each one increases the cost of learning to use Gatsby. Right now Gatsby is almost entirely vanilla React. The only reason I added layouts at all (Gatsby v1 started without it) is that there's enough use cases for where you want to have some containing react components not unmounting on every page change. But sometimes I regret this. We could move to a system for example where we don't force unmount page components so we wouldn't need the layouts in that case. I actually shipped Gatsby v1 with this on accident but it confused people as they'd have components with local state that wouldn't reset when you changed pages (to fix that, they'd have to set a key on the component based in part on location.pathname).

So it seems the least confusing system is having just one top-level layout component and one top-level page component and yeah, using HOC & fragments.

I really appreciate your taking a stab at the nested pages! I hadn't really thought this through carefully until these discussions. I think having lots of people being willing to try prototyping different solutions is the best way for us to arrive at the ultimately best design.

[AbhimanyuAryan]

can we use ReactVR with gatsby?