Full static generated mode

manniL commented 5 years ago

Current state

Nuxt's Static Site Generator (nuxt generate) is growing! I love the static mode when it comes to portfolio pages (or generally, pages that don't include a lot of dynamic data).

Problem

Usually, you use statically generated pages together with a Headless CMS or another external API.

Currently, you can generate the HTML (with static + universal mode) but asyncData calls are still made during client-side navigation, which means that an external API will likely be called on such a static page.

While it's worth here and there to make these calls even after static generation, it's absolutely not needed (from my POV) for the majority of the pages. Instead, the author/developer could simply issue another build (eg. through Netlify) to update the content.

Also you might encounter different content as asyncData is not called on the entry route of you static app. (Going from /b to /a can lead to different content than directly accessing /a)

Proposal

As announced by @Atinux on Vue Toronto (see his talk at 26:18), I want to propose the full static generation option for nuxt generate.

Instead of relying on the API calls, we should inline the response of the asyncData method(s) inside a .json file (per page) when the generate.fullStatic flag is enabled in the nuxt.config.js (the flag name is debatable). Then we can use the .json file as data source without issues.

3rd party modules that apply this approach

https://github.com/DreaMinder/nuxt-payload-extractor https://github.com/stursby/nuxt-static/

YamenSharaf commented 5 years ago

As someone who's working to create a fully static site, this has been an issue for me as it looks like there's no way currently to have a fully static site with a headless CMS. Nuxt has a lot of tools that help with that, but not quite in a way that ties them together to create something as fast as a static site should be.

These are the options I've stumbled across so far:

Relying on `asyncData` in page components

Pros

This is the natural /encouraged approach by Nuxt docs
Each page pulls in its own static content. Refreshing the page yeilds static content without extra AJAX requests.

Cons

When navigating through nuxt-link to pages with async-data function defined, an AJAX call is fired before each page navigation to, again, load the data that is already statically on the page in a needless fashion. I'm no expert, but I guess that entails also shipping the vue engine to bind data coming from the server (also needlessly).

Relying on `payload`

Pros

Nothing I can think of really, since the earlier approach seemed to work fine for pulling data for individual pages.

Cons

More complex to implement.
When navigating through nuxt-link, payload property is always undefined, which means I would have to have an asyncData method that would fire anyway on each page navigation.

Utilizing the `vuex` store and `nuxtServerInit` to get needed data

Pros

No AJAX call is triggered when pulling data from the store (seems fair), even when navigating with nuxt-link.

Cons

Store is basically global and not confined, which means that pulling data upfront. eg, posts, will cause them to load on each page even if not needed, but you do get faster navigation afterward.

Conclusion

I would have had rather be a straightforward way to pull data from an API without the overhead of extra AJAX requests or loading unneeded data.

pi0 commented 5 years ago

Nice writeup @manniL.

For more specific terminology let's call this behavior/mode static and the offline data source files cache data.

Adding some more important notes:

We should consider we have fetch, asyncData, nuxtServerInit and SSR plugins that all should be addressed also can cause side effects like triggering store dispatches that they can also fetch/process data.
~~Cache items have no one-to-one relation with pages or route entries. Wildcard routes and query parameters require different responses.~~ (edited: this is a rare case as we generate one HTML per each)

Regarding implementation and future opportunities I can think of 3 ways:

1. Memorize `asyncData`

This way, during generating we can spy result of asyncData during SSR generate and store it somewhere like .nuxt/dist/cache. Then by modification of asyncData implementation for client-side instead of the original method we import corresponding cache item. This also addresses the need for duplicate HTTP requests.

Pros

Most straight-forward approach.

Cons

We can't guarantee everything is cached. fetch and store may have their own logic.

2. Cache `window.NUXT`

This way, during generate we cache nuxt states into .json files and like method (1), change the behavior of client-side for fetch and asyncData

Pros

Straight-forward and covers store state too.

Cons

The possibility of breaking change for exciting projects.
How to handle non pojos?

3. Cache network requests

This way was the one that me and @Atinux talking about it almost one year ago for static mode. Instead of internal nuxt hacks, we can find a way to spy on network requests and statically cache them somewhere like .nuxt/dist/cache/[hash].json and instead of doing actual requests we can import json chunks.

Cons

Spy part is delegated to official nuxt modules. (However, if changes required to the core, we make them)

Pros

Updating API cache is independent of nuxt generate. We can update it without rebuilds
We have full control over each ajax/http call to be cached. Not caching a black-box
No internal hacks and breaking changes
Transparent caching means making it optional without the need to changes of fetch/asyncData and store actions for offline support
More service-worker spec friendly for caching and PWA like approach.
Works with other modes (@manniL)

pi0 commented 5 years ago

After talking with @Atinux, both (2) and (3) options seem good approaches. Most of the static generated websites do not need complex logic and simply storing NUXT object will be enough but also network level caching has it's own benefits too.

The suggested approach is introducing a new $cache API to the nuxt core which is responsible for storing and loading cache entries. This API may be used directly by users or module like axios. The cache layer may leverage a cache adapter too.

I do believe we can include a really basic default implementation to the core. Adapters should be probably different for server and client (cache.client.js and cache.server.js. Like nuxt/nuxt.js#4574 which made by Sebastien for nuxt-link)

For page level caching during generating full static version, we can add a cache property to the pages or global config (Like transition) that can enable/customize page level caching and probably globally enabled when using nuxt generate --full-static. We can then store ssrContext.nuxt.data to a JSON file that will be loaded instead of calling asyncData and fetch on the client side. This should be also clearly documented that those functions are no longer being called on full static mode.

~~For changing client-side behavior we can use lodash template to replace default asyncData/fetch calls with $cache.get($route) calls (vue-app/template/client.js)~~ For client-side support we can add logics to read cache option from page or global and use $cache.get($route) instead of calling functions. (vue-app/template/client.js)

Atinux commented 5 years ago

I strongly validate @pi0 comment.

This caching system could be used by other modules and inside services too in the future.

Nuxt is build on modules, we simply need to have @nuxt/cache now.

The thing I want to work on is to give more possibilities to modules to customize the Nuxt logic of client-side. This could be done by extending defaults components, or having some hooks system. So Nuxt plugins could also interact more the client & server logic of the default Nuxt logic (middleware, async data, transitions, store, etc)

manniL commented 5 years ago

One aspect we shouldn't forget: Making a page "full static" by actually removing the API calls from the code also means that the underlying API is harder to attack (via DDoS, exploits, ...) as a possible attacker doesn't know where the target API lives.

Of course, Security through Obscurity is no sophisticated security concept at all.

pimlie commented 5 years ago

Could we take this rfc maybe a step further and also include an option to build an all inclusive, single (static) html file? There are times this would be useful, e.g. when you create a demo of something so you can just email a single html file to a client or upload multiple demos to your website without the need for subfolders etc. But also an Electron application could benefit from this when all html/js/cs is loaded at application start-up, or a better example if you want to put a single html file on a usb-stick to hand-out as a promotion.

E.g. in the past I've used a grunt script to achieve this (albeit for a non-Nuxt project). As combining all the css/js/font files resulted in a single 1.3MB html file, I embedded all files zipped & then base64 encoded which reduced the file size to 550KB. (Of course the unzip javascript code and the js for the loading dialog was not zipped).

pi0 commented 5 years ago

@pimlie Nice idea. We may use MHTML format to encapsulate all resources of each page. And it is easier to compress.

pimlie commented 5 years ago

Although mhtml would in theory be better, it lacks the full support which plain html file has (eg Firefox doesnt support it by default). But if all the hooks are in place, we could choose to create modules to generate both mhtml as html.

Atinux commented 5 years ago

Indeed, this could be made with the help of a module @pimlie, this module could:

Overwrite all webpack file/url loader to force include as data64
Find any url that point to a static file and include it's base64
Remove code-splitting to have only one huge JS file?

Also, with 2.4, modules can create sub commands 🚀

DreaMinder commented 5 years ago

I've made a module for this purpose, didn't know that a simillar one already exists. Its very simple, but requires custom code in asyncData like payload mechanism does. Repo: https://github.com/DreaMinder/nuxt-payload-extractor Working demo: https://dreaminder.pro/ru/blog/nuxt-vs-vue-spa-battle (just a sketch of my blog) Maybe you'll find usefull some ideas I used.

manniL commented 5 years ago

Another problem that comes to my mind is queryString. But something like /articles?page=2 won't work properly when you have no server behind it.

We should still think about that

DirkStevens commented 5 years ago

Wonderful thoughts!

We're using Nuxt as the front-end for data from a Headless Drupal. We'd love to be able to render a static site.

These are some of the challenges we face with static rendering:

We've got >40,000 pages managed and maintained by a team of 25+ writers

Most pages are updated regularly several times a year (think: 25 * 40,000 = 1,000,000 updates/year)
Pages are added on a daily basis
Some pages are removed

How and when are re-renders performed of added/changed pages? How do you know about new pages? Removed pages?

How can we let Nuxt know what data changed?

Routes are managed in the CMS by the writers

This is very different from common router scenarios where the routes are known upfront and can be defined beforehand.
Pages can be moved from one route to another
This requires a re-generate of several pages
Routes/slugs can be renamed - this touches all pages that link to that page also
Our routes can not be defined with patterns at all...

How can we let Nuxt know a route changed?

Search pages

We've got search pages with fuzzy and facet bases search that require server side calls
We've experimented with browser bases search on full-data fetches and have excellent results in the UX but problem is to get the data to the client and keep the data current

Some pages combine data from multiple sources

Page A uses DataA
Page B uses DataB
Page C uses parts of DataA and DataB

Ideally the generate takes re-use of data into account. If DataA changes, Page A and C need to be regenerated. If DataB changes Page B & C need to be regenerated.

We're not familiar with the deep insides of Nuxt but would love to help.

Cheers!

DreaMinder commented 5 years ago

@DirkStevens Wow, sounds like worst case scenario for static-generated project =) I'm not nuxt contributor, but I think complicated invalidation logic varies with every project and can't be included in framework core. If you want speed up your project, I'd suggest to go for nginx-cache or varnish. There are ways to communicate with between nuxt and proxy-cache to invalidate updated content, it would be much better reliable solution.

Just imagine that you changed navigation item in your layout and you now have to generate 40k pages, because every page contains navigation.

pimlie commented 5 years ago

@DirkStevens You'd probably should read this question: https://github.com/nuxt/nuxt.js/issues/2370 and the nuxt generate docs in general.

DirkStevens commented 5 years ago

@DreaMinder @pimlie Thanks for your thoughts.

True, invalidation logic should happen in the project and not in Nuxt. For large, high-impact production static sites it would be good if Nuxt exposes events that let us trigger partial re-generate of items that we mark as invalid.

Right now we use Akamai CDN. @DreaMinder Could you point me to information about proxy-cache invalidation with Nuxt?

DreaMinder commented 5 years ago

@DirkStevens sorry, no. Maybe I'll make an article about it, it is complicated. Shortly, nginx-proxy-cache (or cloudflare) respond with cache by default but checks in background if content is updated with limited interval. I forgot how this strategy is called...

sebtiz13 commented 5 years ago

I'm just starting on nuxt, but for my blog i have create an modules to download the list of my articles and I do my treatment in hook "build:before" for save that in json file.

After in pages I import this json file and return the part of data which interest me in asyncData. Currently I have only 5 articles on my blog so I do not know the impact on the performance of generation and its most a proof of concept for the introduction to the static site for me.

But i think its not bad because the file is transformed in js file with hash by nuxt, so the browser can keep it in cache.

I know this solution its not adapted for big site, but I think we can create chunked file with the specific data for each pages.

Finally Its true that it would be nice if nuxt managed the creation of these files through asyncData and this function its only called on server side.

yann-yinn commented 5 years ago

My two cents

Nuxt is actually an awesome Static Site Generator and i built several sites it as a static generator.

My guess is that a solution like Gridsome will probably soon get most of the hype concerning JAMStack with Vue because :

It is focused on JAMStack only, so it will probably move much faster in this direction
It will offer, out of the box, some plugins to pull data to generate pages (yaml, wordpress, contentful etc), so it will probably work faster for beginners
There is a GraphQL layer to get the data, people seems to love the idea and tha's part of what made Gatsby so successfull (mixing the good words together : JAMStack + React + GraphQL, bingo ^^').

I would say the most important things for the users is to have plugins to fetch data from sources in a easy way, out of the box.

For example : today, how do you fetch data from markdown files to generate html files ? Which is the best way to create a "Hugo Like" or "Jekyll Like" static site ? This is not something that is easy to find. I began to code my own package for that, nuxtent was a solution but seems not to be maintained anymore.

The asyncData / JSON files issue is more a technical issue than a way to vastly improve the Developper experience when generating static pages with Nuxt.

manniL commented 5 years ago

@yann-yinn Thanks for the feedback ☺️

A few words on your post:

It [Gridsome] is focused on JAMStack only, so it will probably move much faster in this direction

Could be possible, but doesn't have to be. We are also concerned with JAMstack as you can see 😋

It will offer, out of the box, some plugins to pull data to generate pages (yaml, wordpress, contentful etc), so it will probably work faster for beginners

True, that's something we can start with / focus more on. Easier integration with Headless CMSs. There are some nice initiative (e.g. this module or this module) out there.

Also, more tutorials on that topic would be great, I agree.

The asyncData / JSON files issue is more a technical issue than a way to vastly improve the Developer experience when generating static pages with Nuxt.

I don't agree with that. A "full static mode" would increase the performance of the page and would lead to less mismatching content (as explained in the intro post).

yann-yinn commented 5 years ago

I don't agree with that. A "full static mode" would increase the performance of the page and would lead to less mismatching content (as explained in the intro post).

I agree a "full static" generation without additional asyncData call on client Side would be cool and also help to have more "atomic deploys" , which is part of JAMStack philosophy. My concerns about source plugins and comparison with Gatsby-like static site generators should be in another issue than this one.

cesasol commented 5 years ago

I do think that the static or event the ssr mode could be improved, this article points that website made with techniques like hydrating are actually bad but could be improved with partial hydrating. In my case I often find myself looking at big page components that only render static html but takes twice the time to get to an interactive state (hydrating) for the huge code that is makes, so far I managed two options, one is to use a functional per view component that renders only static html, the second (currently named nuxtent) is to have the html or markdown in another file and use it as an api.

Felwin commented 5 years ago

Please allow a mixed mode, I might want to get some data in static (fetched from my API only when generated) but still get some other data dynamically. My actual use case is an eCommerce website where I don't want to make an API call to fetch my SEO texts (I can re-generate when modified), I would like to have them static, but I still want to dynamically check for products availability and prices. If there was a new function like asyncData that only fetch data at server rendering while asyncData behavior stay as it is now would be perfect.

yann-yinn commented 5 years ago

@Felwin Why not call your product availability API inside the mounted lifecycle ? it will be call only from client side, so it will be "dynamic". You can open an issue about that if needed, this is not related to this issue ( being about not calling asyncData twice) :)

Felwin commented 5 years ago

@yann-yinn I could do the fetching in mounted indeed, but then It won't be loaded before the page render during the nuxt loading bar.

Anyway if not possible to do a mixed mode I would prefer full static generate mode rather than the actual behavior.

yann-yinn commented 5 years ago

Hello @manniL @Atinux @pi0 ! Any updates on this ? Would love to see this feature in Nuxt ! 💛 .

maikueo commented 5 years ago

although not ideal because it needs to reload all assets on each page load and anything in your vuex store will be gone, and maybe forget about page transitions, but for those who just want a quick way to get a full static site in the current version of nuxt, you could try switching out all nuxt-link for old fashion 'a' tags. This probably won't be good for everyone, but some people might find it preferable to current hydration issues.

thomasaull commented 5 years ago

@maikueo You'll basically loose all the good things about SPA's in this case (fast navigation, transitions, …) 💁‍♂️

maikueo commented 5 years ago

@thomasaull I completely agree :). You would also lose support for some of the cool things nuxt-link does, like prefetch and setting active class. I think the solutions mentioned here sound really good.

robertpiosik commented 5 years ago

Hello everyone. I've just used DreaMinder/nuxt-payload-extractor to achieve what's discussed here. Please try to jump between projects which are located below the slider to see how the UX is affected by the slight delay caused by the request for pregenerated on build step json file. This operation is much more effective that regular API call as the whole application is served statically from the CDN.

I see also a room for the loading speed improvement by preloading any payload data file of pages related with nuxt-links to currently visited route.

Would love to see this implemented similarly in the core to cut off the little dependency.

altryne commented 5 years ago

Been trying to find a tutorial or a way to do a full site static generation with Nuxt as well. The VUEX approach makes sense but it's pretty heavy and not very simple to achieve.

thomasaull commented 5 years ago

@altryne I was successful with https://github.com/DreaMinder/nuxt-payload-extractor. In case you're interested I can share an example

Atinux commented 5 years ago

I started extracting the module and I am actually using it for the documentation: https://github.com/nuxt/nuxtjs.org/blob/master/modules/static/index.js

I plan to:

Add generate.static option (default to false and true for Nuxt 3), when this option is activated, @nuxt/generator will extract the payloads.
Set the option to true with nuxt generate -s or --static
Add an option static: false (only for process.static) to disable payload fetch and call asyncData/fetch
Enhance nuxt-link to prefetch also the payload when visible on viewport (with low priority) and PageComponent.static !== false, once the Page bundle is loaded.
Use our crawler in union with generate.routes (crawler already working on https://github.com/nuxt/nuxtjs.org/blob/master/modules/crawler/index.js) and move it inside @nuxt/generator

Make a PR and hopefully land it for v2.9.0 😄

gridsystem commented 5 years ago

@Atinux I have put together some workarounds for this myself in the past, glad to see that you have approved a direction to take.

After skimming this thread I would like to clarify that I have understood correctly, can you please confirm?

asyncData will hopefully have native support for disabling repeated API calls in 2.9.0
fetch will also receive the same treatment
nuxtServerInit is only populated during the generate task, and there are no repeated API calls during normal production use of the generated site, so no changes are necessary.

Atinux commented 5 years ago

When using nuxt generate indeed, nuxtServerInit, fetch & asyncData won't be called but Nuxt will use the payload when generating the page.

huttarichard commented 4 years ago

@Atinux when can we expect to see this feature? Very excited about this one.

yann-yinn commented 4 years ago

So are we at Blogify (http://blogify.io) ! A full static blog generated with Nuxt will be pure dope and will avoid some unnecessary http request to our API. Moving fully from Wordpress blog to a Nuxt blog \o/ (well , it works quite well already but still can't wait for full JAMstack mode with my favourite front-end techno)

0x7357 commented 4 years ago

Hello,

I would welcome the possibility that so-called "payloads" are embedded directly into the generated source code. I can well imagine that this is not very easy to realize. ;-)

An idea would be for example:

{
  inlinePayloads: true
}

... and then loading the payload (for example)

<script src="./payload.json"></script>

... is replaced by ...

<script>
  const payload = {};
</script>

Best regards Danny Endert

manniL commented 4 years ago

In case we can implement the option to inline content as @DannyEndert suggested, it’d be ideal to make the behavior customizable, i.e. only inlining small payloads, but also giving the options to inline everything or nothing

robertpiosik commented 4 years ago

@manniL it's not cleaner to use for this purpose nuxtServerInit action? In my opinion asyncData payloads should always reside in json files which could be downloaded or not, depending on routing. Just to stand in place of currently made http request.

Again, for every need of populating application with data on static generation step currently available nuxtServerInit fits perfectly. We shouldn't combine everything together, unless avoiding vuex usage is really necessarily.

0x7357 commented 4 years ago

@robertpiosik Sounds legit.

But for me (personally) it's better to have asyncData inline instead of an external file. I kinda hate request-making static websites - even if it's just a small json file. CSS is working properly and since the asyncData is quite a part of the static website, it should be (just my opinion) inlined.

Now you could say "the vendor files aren't inline aswell" and I'd love to say something against this. I see the payload as "the content" and the static website should "contain" it. Sure, it's not the best reason - but I'd love it. :-)

sustained commented 4 years ago

This sounds really cool and I'm looking forward to playing with it.

robertpiosik commented 4 years ago

@Atinux

Make a PR and hopefully land it for v2.9.0 😄

Does it land finally?;)

uptownhr commented 4 years ago

it would be cool if static can be defined inline in asyncData or fetch as part of the context. Proposal.

async asyncData ({app: {$airtable}, route, static}) {
      const table = $airtable('listings')
      const record = await table.find(route.params.id)
      return static({
        data: record._rawJson,
        record
      })
    },

If the method static method is called, the payload is cached as part of the codebase on generation.

manniL commented 4 years ago

@uptownhr Turn static mode on/off per page will be possible, but only caching a partial result of asyncData could lead to problems though, as it'd be hard to partially "recall" asyncData and leave out the static parts.

gridsystem commented 4 years ago

@Atinux Do you have any idea of when this might make it into a release?

gridsystem commented 4 years ago

I hate to keep bumping this but I’m not really sure where else to track progress. Is there any news @Atinux or is there somewhere better I can follow the team’s planning process?

timstapl commented 4 years ago

Any update on this?

gridsystem commented 4 years ago

@manniL @Atinux I just want to say, if you have a branch you can push which the community can finish off, I think you have at least 18 people who could potentially take it the rest of the way to a PR.

I started extracting the module and I am actually using it for the documentation: https://github.com/nuxt/nuxtjs.org/blob/master/modules/static/index.js

I plan to: … Make a PR and hopefully land it for v2.9.0 😄

I personally am a little frustrated when this happens in an open source project. Nuxt is clearly open to input from the community, but a core developer has taken the lead on a feature without sharing their code (or progress). Other contributors could have proposed solutions and raised PRs in this time, but have not because a core dev has had no transparency with their working.

I don't think anyone here is unhappy that you're taking a while with this, or even if you've decided not to pursue it. If you say you're not going to, then we can all embrace third party modules or custom code for this feature and move on, confident that the code won't be deprecated by a suddenly released core feature. Or use Gridsome, which seems to have embraced this concept.

If you're still RFCing, I think this raises an important topic - it might be useful to look at how this could have been handled differently. Are there any processes which could have been put in place to allow for the community to finish this code? Is there a platform or a GitHub feature for maintaining transparency with feature progress which could be utilised? Is there a way for the community to vote on and prioritise features which the core team wishes to plan and implement themselves?

Atinux commented 4 years ago

Hi @gridsystem

I am still working on it, it took me more time than planned (very busy lately) and avoiding breaking changes is pretty tough, the PR implementing it is here: https://github.com/nuxt/nuxt.js/pull/6159

gridsystem commented 4 years ago

This is fantastic. I think the solution to my final paragraph is only to link PRs to RFCs!

If I put my PM hat on, good next steps (if you have time) would be to

Braindump your notes as a comment on the PR
Add quick summary of progress as comment to PR
Add quick overview of why tests are failing as a comment to the PR
Update your PR description (add any checklist items and check off complete items)

And then throw it into the wild for the community to help get to the finish line? This is such a great feature! Thanks so much for working on it.

nuxt / rfcs