Allow full client-side rendering + use pre-rendering for SEO bots

[bloodf]

From vue-storefront created by michael-bouvy: vuestorefront/vue-storefront#4089

What is the motivation for adding / enhancing this feature?

SSR requires having a Node.js instance rendering pages, which can be a pain point in terms of availability and/or scalability of an app, and also requires CPU resources ($$).

Google officially recommends dynamic rendering for fully Javascript rendered websites: https://developers.google.com/search/docs/guides/dynamic-rendering:

• Serve fully static content (page structure) to regular users: dist/ folder could be served for instance from a CDN
• For SEO bots (detected based on their User Agent), make a pre-render call and cache the result, using https://github.com/GoogleChrome/rendertron for instance

What are the acceptance criteria

• [x] Allow disabling SSR
• [x] Serve full front app for regular users requests
• [x] Provide a working reverse-proxy configuration (or Lambda script) to handle pre-rendering

Can you complete this feature request by yourself?

• [ ] YES
• [x] NO (partially, could help on reverse-proxy, testing and documentation)

Which Release Cycle state this refers to? Info for developer.

• [x] This is a normal feature request. This should be available on https://test.storefrontcloud.io and then after tests this can be added to next Vue Storefront version. In this case Developer should create branch from develop branch and create Pull Request 2. Feature / Improvement back to develop.

[bloodf]

That’s actually very interesting idea @filrak @andrzejewsky @gibkigonzo

[bloodf]

@andrzejewsky, we are actually struggling with NodeJS on high load. We need a solution for VSF 1.x, we can't move to 2.x when it's gonna be ready. Our project has only one year lifetime. It's impossible to sale one more migration project to my stakeholders. You guys need to help us. Thank you.

[bloodf]

I totally agree with this feature and think that it's high priority. SSR takes a significant amount of resources and if most of the traffic is uncached page hits, it really makes an impact on performance and time-to-first-byte.

I'm finding out that using the production server to build Vue Storefront is inefficient. One idea that came to my mind (I'm not sure how practical it is), is prerendering and caching a list of high-traffic URLs predefined in the config at the time of building Vue Storefront.

There are other alternatives, such as https://prerender.io/

Your thoughts?

[bloodf]

@pkarw I tried the built-in Vue Storefront generator via yarn generate all, and it consuming more and more memory until the system runs out of free memory. I set the -s size option to 1 just to test and the same happens. Stopping the script frees the memory.

The script halted in the beginning due to JavaScript running out of heap memory, so I had to add the --max-old-space-size=8192.

Am I missing something or is it a bug? https://i.imgur.com/pxl7KNo.png

[bloodf]

Thanks @themreza! But is is necessary to generate all pages, and not only layouts (ie. home, category, product, etc.) ?

Contents would be eventually loaded using CSR from AJAX results.

[bloodf]

Pretty loosely but still related: #3899, #3897

[bloodf]

@michael-bouvy The main issue is that when crawlers reindex the site, they will load a bunch of pages. If those pages are uncached, SSR will consume resources. In parallel, it can hog up the resources fairly quickly.

Of course caching all pages at once is a long process, but it's still a way to offload the SSR strain. I was thinking about providing a list of frequently visited URLs to cache. Maybe that list could be auto generated based on page views.

[bloodf]

we should definitely try this out; within 1.11 there is a static page generator included that might be an option @themreza however it’s not yet used anywhere on production

We definitely should give this feature a try

[bloodf]

@themreza the generate process is currently ingle thread, with no proper memory management i guess it should be decomposed so the single page generation is a single process (or reset the process after X pages so the memory can be freed) then we need to implemen t the queues; Then we need to maintain the page re-generation/invalidation when anything on the page (prices, stocks) changes; that's why it's not production ready yet - but it's a good foundation I guess

[bloodf]

We can create new entry file based on client-entry.ts and new build:prerender script. In this new entry we need to remove parts that assume that there was ssr (window.__INITIAL_STATE__, store.dispatch('url/registerDynamicRoutes'), etc) and enable asyncData on first load (here it's disabled because we assume that there was asyncData on ssr https://github.com/DivanteLtd/vue-storefront/blob/master/core/client-entry.ts#L82 ). And with that changes (in theory) we would have SPA app. From that point we can use prerendering to generate static pages.

[bloodf]

@themreza what is the purpose of VSF's yarn generate all command? Generating a fully static layout?

[bloodf]

another solution is just using caching layer for SSR pages, render once, serve many times (eg. Redis)

[bloodf]

@andrzejewsky yeah actually we have it out of the box. I’m not sure if this external SSR service will provide us with expected performance results over VueSSR + cache we already have; however it’s worth trying and I heard that @themreza is working on kind of PoC?

I guess that we should have this second option available as a server module or config option. @gibkigonzo notes will be very helpful in implementing this

Then we maybe should have another Config option - for letting users to disable / enable SSR for specific user agents.

I think this could be a use full thing as the crawlers don’t necessary needs always up to date prices and stocks (they’re updating the index once per day usually anyway); Then another option to disable vuessr at all (so core/scripts/server.ts will serve just the dist/index.html) nothing more) so we can cache the bot related HTML more aggressively

So maybe setting TTL per user agent (in edge cases set to 0 will disable the cache) + having another rendering engine (prerender or any other) will be a perfect solution?

The last option is to use the page generator (embedded from vs 1.11) or prerender.io used as a kind of crawling service - the issue here is caching / invalidation (this problem doesn’t exist in our ssr caching solution as the invalidation is provided directly from the indexers)

Let’s do some POC! I really like this idea.

Reference materials: https://docs.vuestorefront.io/guide/cookbook/checklist.html#_2-ssr-output-cache

[bloodf]

Great job @gibkigonzo!! Will give this try this asap with @jonathanribas 👍

[bloodf]

I've made simple spa implementation https://github.com/DivanteLtd/vue-storefront/pull/4120 If I find time, I will check if we can generate at least the layout. Here is vsf on netlify https://peaceful-chandrasekhar-e7a2a4.netlify.com/ :) I've checked this site on https://render-tron.appspot.com/ (https://github.com/GoogleChrome/rendertron) and it looks ok. Probably we can stop here and just made middleware

[bloodf]

@michael-bouvy Sure, that's a possible part of the solution, which is what prerender.io is offering. The other half is reducing the SSR load by precaching known URLs that have a high non-bot traffic.

[bloodf]

@michael-bouvy https://github.com/DivanteLtd/vue-storefront/blob/f3c218d7a202718915d13a87f09ee57a04396c0b/docs/guide/basics/static-generator.md

@pkarw I see, I will give it a shot. Also, is it possible to save the results in the Redis cache instead? Since cache invalidation is already implemented there.

[bloodf]

yeah it is.... not to sure how hard that will be to implement in VSF 1.x but the idea sounds interesting : )

[bloodf]

@themreza indeed, but this would go along with a crawling bot, caching SEO pages with a long TTL (ie. 30d)

[bloodf]

@gibkigonzo awesome job! Having this rendering middleware you mentioned in the PR and we have it!