Server side rendering discussion

ashtuchkin commented 11 years ago

I would like to create a module that will make server-side rendering in node.js possible. I've done this before in my own project and wanted to discuss technical details.

So, the goals of the module:

Make server return fully rendered html on first request, then move to client side rendering seamlessly.
Support html5 routing defined through $routeProvider on the server.
As little changes to client-only rendering as possible.

How do I want to do that:

Module will be a Connect/Express middleware that you can app.use().
Use jsdom to create & work with DOM.
To enable client-side rendering, developer will need to provide all html code as templates and use only <ng-include> and <ng-view> in main html file.
On request, a new DOM is created and these <ng-include>-s are filled with rendered html. When ready (see below), the full DOM is serialized and sent to the browser (or crawler). In browser, the contents of <ng-include>-s are discarded and re-rendered, creating all services and controllers, binding all needed event handlers.
When rendering on server, $http requests are re-routed to use the same server.
If $routeProvider didn't match anything, the middleware will yield back to Express to serve later middleware or a 404 page.

Problems to solve:

jsdom will need to be extended with sufficient userAgent and location/history APIs.
The 'page ready' event is something to think about. I could use the $browser.notifyWhenNoOutstandingRequests() internal API, but maybe it could be better to introduce a $ready event on $rootScope.
I'll need a way to intercept angular bootstrapping to be able to catch this 'page ready' event. It is currently done in ngScenario, but I think it could be better (for example, it doesn't support asynchronous bootstrapping).
If we find a way to cleanly intercept angular bootstrapping on the page, we could have a single angular codebase residing in node process, serving all pages by just creating injectors. This should make page serving performance comparable to more traditional template-based servers.
$routeProvider will need to be extended to provide an event when no routes are matched. Currently it's an otherwise clause which usually redirects to base, but in server environment it's bad practice. Preferably even on the client side if a route is unknown, we should redirect the browser to new url, as the server can have something else there (statics, etc), or 404 page.
It would be nice to cache all $http requests that are done on the server, and package them together with html, so that first render in browser would not ask for the same information again.
I don't know how to test this whole endeavor. I can make e2e and unit tests to prove that jsdom is nice behaving browser, but I'm not sure what to do about ensuring that the browser will always get correct state when served with this method.

So, what do you think? Is it viable? Did you plan to do something similar?

dai-shi commented 11 years ago

+1. I would even expect to continue server-side rendering afterwards, for JavaScript-disabled browsers and IEs.

dai-shi commented 11 years ago

So, I made a connect middleware for server-side rendering. It's actually independent from angular.js, but angular.js is my primary target. https://github.com/dai-shi/connect-prerenderer

ashtuchkin commented 11 years ago

Interesting, I've been thinking along the same way. Do you think the timer approach is viable in production envs? Also, does it work with angular routing?

Alex

On 11.03.2013, at 7:41, Daishi Kato notifications@github.com wrote:

So, I made a connect middleware for server-side rendering. It's actually independent from angular.js, but angular.js is my primary target. https://github.com/dai-shi/connect-prerenderer

— Reply to this email directly or view it on GitHub.

dai-shi commented 11 years ago

I have been thinking the use of PhantomJS, but your post and https://github.com/tmpvar/jsdom/issues/380 pushed me to use jsdom. The timer approach is a fallback, and the client-side javascript code (that runs on the server-side :-) is expected to notify the end of rendering. I haven't tested with angular yet. It won't work without a modification. Some tweaks are needed.

dai-shi commented 11 years ago

I finally made it work with AngularJS. https://github.com/dai-shi/connect-prerenderer At this point, only basic e2e tests are available, but I see it feasible. I had to patch angular.js so that it keeps templates for interpolation after compilation and that it deletes duplicated ng-repeat elements before compilation. It's more or less a hack right now. The documentation is weak, but I can help if anybody is interested in using it.

lmessinger commented 11 years ago

@dai-shi thanks! this looks so very cool. i'll be looking at it for our project. I'm using ui-router (https://github.com/angular-ui/ui-router/)

donaldpipowitch commented 11 years ago

Any update on this? Would be great to use Angular like you would use Jade as a view engine for Express.

dai-shi commented 11 years ago

My project https://github.com/dai-shi/connect-prerenderer is getting good, still not mature. It works with ng-repeat. There's one open issue. Bigger test cases would be required. Would be worth trying until Angular 2.0 be ready.

FloNeu commented 11 years ago

Hi, guys! Just stumbled over this discussion... Very intresting, as i am working on a project-setup using grunt, express, angular.. for which i also want a server-side template-rendering fallback. Exited to try your prerenderer. Many thanks

btford commented 11 years ago

As part of our effort to clean out old issues, this issue is being automatically closed since it has been inactivite for over two months.

Please try the newest versions of Angular (1.0.8 and 1.2.0-rc.1), and if the issue persists, comment below so we can discuss it.

Thanks!

donaldpipowitch commented 11 years ago

It's still an issue.

dymk commented 11 years ago

@btford, the issue should be re-opened; it's still a valid issue in 1.0.8 and 1.2.0-rc.1.

FloNeu commented 11 years ago

Ideas on this topic? I assume that the purpose of running angular on a server is to serve some fallback functionality. I am using angular in combination with nodejs/expressjs server. Usually my angular views get prerendered and then combined with the wrapper directly served from a jade-template with some live-server data i use as angular no-js fallback (like navigations - with std. routes - these are replaced with angular routes, if present).

This works create for angular directives that replace the contents of tags ( ng-bind, ng-inclued etc. ). The thing is that I am running into problems when i want to do the same thing with html-attributes, because i render the attributes content from jade, not an angular-placeholder.

So i am currently playing around with these possible solutions ( a combination of those ):

Only use Jade for convenience and not for logic. Run angular on the server, only use angular-placeholders and serve fallback-pages from this instance. PRO: Less more readable code as there are no jade-conditionals, loops, placeholders etc. in the jade-Templates. Fallback for Angular-Routes CON: Doesn't solve prerendered-attributes issue. Is limited to NodeJs-environment where either a headless-browser/JSDom can run.
Do as now and write a angular-directive for marking attribute-contents to replace with angular-placeholders. PRO: Easy, Graceful degradation, can be used with any template-engine CON: 'Logic-Doublication' in Jade and Angular.

All the best, Florian

donaldpipowitch commented 11 years ago

I would like to reuse the complete logic of my AngularJS on the server side e.g. routers, etc. and render the whole site on the server. If the user has JavaScript activated the site gets initially rendered on the server and after that the site behaves like a SPA (no flash of content, no initially spinners, easy SEO, etc.).If the user has JavaScript deactivated the site behaves like a normal static page - except my templating engine on the server is Angular and not Jade, EJS, etc. and I describe routes in Angular and not Express. I think that would be graceful degradation.

FloNeu commented 11 years ago

I see... I do a similar thing with my routing... i define all my projects paths, routes, navigations in a node config module. From this i generate my express and angular routes. The outcome is similar/same to yours. The problem i encountered with reusing angular-route definitions is that i need more complex routing mechanics for the server side. Like auth plugin etc. that i can't define properly in angular routesJs-file, but i still want to define them in one place. So i went more the sharing server-resources with angular way, then the other way around. When i am back @ a stable state i would be interested in exchanging and comparing our solutions. Further i don't want to loose Jade ( but maybe the prerendering of angular-views ), as it made me addicted to it's style of html-templating in hours :)

donaldpipowitch commented 11 years ago

Auth is a problem, yes. A RESTful API helps a lot to hide private data and with some additional redirecting it should be possible to prevent users from seeing private views. I would treat Jade just as a different Syntax for HTML e.g. use Jades meaningfull whitespace to avoid forgotten closing tags but don't use mixins, iterators, etc. because you can use Angulars directives for that. So it works on client, too.

lmessinger commented 11 years ago

btw, have you guys looked at https://github.com/dai-shi/connect-prerenderer? I've used his techniques (patched https://github.com/dai-shi/connect-prerenderer/blob/master/test/server/public/angular.js) together with ui-router and phantomJS to produce snapshots on the server for SEO purposes. the good thing is that that these could be bootstrapped a-la Rendrhttp://nerds.airbnb.com/weve-open-sourced-rendr-run-your-backbonejs-a/

On Tue, Aug 27, 2013 at 5:24 AM, donaldpipowitch notifications@github.comwrote:

Auth is a problem, yes. A RESTful API helps a lot to hide private data and with some additional redirecting it should be possible to prevent users from seeing private views. I would treat Jade just as a different Syntax for HTML e.g. use Jades meaningfull whitespace to avoid forgotten closing tags but don't use mixins, iterators, etc. because you can use Angulars directives for that. So it works on client, too.

— Reply to this email directly or view it on GitHubhttps://github.com/angular/angular.js/issues/2104#issuecomment-23323116 .

Lior Messinger 1-646-3730044 A.lgorithms.com

FloNeu commented 11 years ago

@donaldpipowitch You are totally right about the Jade vs. Angular thing... as with this prerenderer approach it is the only way to stay DRY. Already started to rebuild my templates...

@lmessinger Hmm... I will give that connect-prerenderer a try... plan is to use it in combination with a directive that reinserts angular placeholders to html-elements that have been prerendered, haven't found a better solution to handle this (Do you guys understand what my problem is with the attributes?)... I am still trying to wrap my head around this approach... I have a running express app-server which uses a headless-browser to create a live dom on the server side. A http-request is received via the no-js routes and forwarded to the headless-browser to call the page in it and create a snapshot of the result to return to the client. The problem i see with that is that i can only have one page/dom at a time and if a second request comes in i have to wait until the last request is finished before i can build ab the second Snapshot ( is that correct? ). Any experiences how this scales? Or am i just plain wrong.

Will try to get it running over the next few days. Enjoying the discussion! Thx guys...

lmessinger commented 11 years ago

Florian Hi

First the nice thing about connect-prerenderer is that it has the Angular code you mentioned, that allows to bootstrap an application from the snapshot. ( This is presented as a patch to Angular itself). Bootstrap means, that when the user clicks the link in the search engine results page, she will get an html of the snapshot. The browser will then present the html. In the background it will download the code of the app. It will then start the app and connect it to the html so that further interaction with the page will be done thru the app. The html is decorated so that Angular can know the directives. The connect-pre renderer project waits for a user click to bootstrap the app. I used a patch on ui-router, that allowed me to bootstrap without wait. In essence, I did not allow partials to be attached to the Dom, until the url has changed.

About the flow you mentioned, another approach is to pre render, using the technique above, the indexed urls ("pages") and save them as html files on the server. Once the server seemed that the request is from a search bot, it will serve these files to the bot.

Hth Lior On Aug 29, 2013 12:39 AM, "Florian Neumann" notifications@github.com wrote:

@donaldpipowitch https://github.com/donaldpipowitch You are totally right about the Jade vs. Angular thing... as with this prerenderer approach it is the only way to stay DRY. Already started to rebuild my templates...

@lmessinger https://github.com/lmessinger Hmm... I will give that connect-prerenderer a try... plan is to use it in combination with a directive that reinserts angular placeholders to html-elements that have been prerendered, haven't found a better solution to handle this (Do you guys understand what my problem is with the attributes?)... I am still trying to wrap my head around this approach... I have a running express app-server which uses a headless-browser to create a live dom on the server side. A http-request is received via the no-js routes and forwarded to the headless-browser to call the page in it and create a snapshot of the result to return to the client. The problem i see with that is that i can only have one page/dom at a time and if a second request comes in i have to wait until the last request is finished before i can build ab the second Snapshot ( is that correct? ). Any experiences how this scales? Or am i just plain w rong. **

Will try to get it running over the next few days. Enjoying the discussion! Thx guys...

— Reply to this email directly or view it on GitHubhttps://github.com/angular/angular.js/issues/2104#issuecomment-23469967 .

jiridanek commented 10 years ago

There is a blog where the author manages to get prerendering working with jsdom and some other library to mock JSON requests. https://github.com/ithkuil/angular-on-server/wiki/Running-AngularJS-on-the-server-with-Node.js-and-jsdom . I do not see this mentioned anywhere on the page, so here you are.

daslicht commented 10 years ago

This also seams to work: https://github.com/daslicht/express-phantom (my fork to make it work with express4 )

kevinSuttle commented 10 years ago

Yes, I'd love to see server side compilation in Angular. There are some surprisingly powerful implications for progressive enhancement.

lmessinger commented 10 years ago

Kevin: could u elaborate on the implications? im curious

On Wednesday, May 21, 2014, Kevin Suttle notifications@github.com wrote:

Yes, I'd love to see server side compilation in Angular. There are some surprisingly powerful implications for progressive enhancement.

— Reply to this email directly or view it on GitHubhttps://github.com/angular/angular.js/issues/2104#issuecomment-43845864 .

Lior Messinger 1-646-3730044 www.Toyify.me http://www.toyify.me/ www.Lgorithms.com http://A.lgorithms.com

kevinSuttle commented 10 years ago

@lmessinger See: https://github.com/angular/angular.js/issues/5723#issuecomment-43906582

FloNeu commented 10 years ago

This module also looks promising... https://github.com/meanjs/mean-seo

hiravgandhi commented 10 years ago

Working at a large e-commerce company, I have seen us trying to use hybrid approaches like those used by Twitter and Airbnb (see http://nerds.airbnb.com/weve-launched-our-first-nodejs-app-to-product/ and https://blog.twitter.com/2012/improving-performance-on-twittercom) to ensure consumers spend less time till first interactions. Companies like these are using a hybrid approach to render SPAs: first, generate pure HTML content so that rendering time is pretty constant across all browsers and then second, use a client-side framework (in Airbnb's case, Backbone) for subsequent loads. I am curious to see if this is in Angular's development timeline as an integrated solution with Node.js or some other server side platform.

daslicht commented 10 years ago

hiravgandhi: This is the perfect approach for me either.AngularJS + ExpressJS and first page rendering on server would be lovely.

daslicht commented 10 years ago

Which is faster: Download App to Client Render there ~or~ Render First page call with PhantomJS on the server and any further things on the client ? ! When Rendering pages with PhantomJS on the server are we able to use the AngularJS after rendering of will that just be a static HTML Snapshot? If so, how to make the returned snapshot full functional

hiravgandhi commented 10 years ago

@daslicht - Running PhantomJS seems like unnecessary overhead. A headless browser rendering content seems like an inefficient solution when ideally, some server side platform like Node.js should be able to interpret and render a Javascript SPA built with Angular.js given some type of package.

daslicht commented 10 years ago

@hiravgandhi Yeah that would be true if we could reuse Models hand Controller of AngularJS in Node :) Possible how ?

Its possible with https://github.com/rendrjs/rendr

But that's Backbone

kevinSuttle commented 10 years ago

Does this affect anything?

http://googlewebmastercentral.blogspot.com/2014/05/understanding-web-pages-better.html

hiravgandhi commented 10 years ago

@kevinSuttle - It affects one of the two problems of using a client-side framework. The problem is that we can't control the speed of initial loads. It would be useful to be able to render Angular on the server side for slower clients or richer content sites, at least for the first page, to get the user engaged ASAP as opposed to waiting for content to load, especially if they have a slower computer.

daslicht commented 10 years ago

morgs32 commented 10 years ago

I've been pondering a module converting angular to ejs. Is that a reasonable direction?

apparentlymart commented 10 years ago

We've been working on a solution to this problem at Say Media: https://github.com/saymedia/angularjs-server

The methodology is to run Angular against jsdom and then exploit Angular's dependency injection to override certain core services to make them work better in the server environment.

As well as generating static HTML on the server it can pre-resolve the route on the server and deliver the route locals inline in the initial page view, so the first page can render without any further data round-trip. Optionally it can also do route resolution on subsequent navigation via a single HTTP call to the server.

Your average Angular app doesn't perform too well on the server (jsdom is slower than a browser, so it'll magnify any perf problems you already had) so we mitigate it by running Varnish in front. A variant of this code is running on our production sites.

jeffwhelpley commented 9 years ago

So, the problem with running Angular 1.x apps on the server fundamentally is that you either need to use a headless browser or a framework built on top of a headless browser (i.e. like @apparentlymart's Angular Server project) which can be hard to make performant OR you can use something like the Jangular library I created (https://github.com/gethuman/jangular) which only supports a subset of Angular and you need to be really careful about how you implement it (but on the plus side it is extremely fast).

Fortunately, there is a much better solution that is in the works in Angular 2. I know that doesn't help with your Angular 1.x apps right now but if you are interested in learning more about what is coming, tune in to the livestream for our AngularU talk next week: https://angularu.com/ng/session/2015sf/angular-2-server-rendering

Btw, @kevinSuttle regarding the Google search crawler indexing HTML rendered on the client side, just realize that:

Google is getting better at it, but it is still not perfect. Server rendering is still a safer bet to guarantee what will actually get indexed.
Other search engines are not as good as Google at this. Also, you have other services like Facebook link preview that is purely server side only.
Indexing is different than ranking. Even if Google indexed your content perfect, that doesn't mean it will rank. And one of the biggest on-page factors for ranking is:
Initial page load performance. Even if you don't care about search ranking initial page load performance is really important for good UX, especially on low powered mobile devices.

marcysutton commented 8 years ago

Just wanted to add that Google has since deprecated their AJAX crawling scheme, making server-rendered apps even more important. https://googlewebmastercentral.blogspot.com/2015/10/deprecating-our-ajax-crawling-scheme.html

gkalpak commented 8 years ago

I'm all for having server-rendering capabilities, but how does deprecating Google's AJAX crawling scheme make this even more important ? (In case that wasn't clear, I'm genuinely curious :smiley:)

marcysutton commented 8 years ago

More important if you're currently only serving up a client-rendered app with no static HTML on the server. From that post: "Since the assumptions for our 2009 proposal are no longer valid, we recommend following the principles of progressive enhancement." How often do you see progressively enhanced Angular apps?

jaydiablo commented 8 years ago

In the same article though:

Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers.

Yes progressive enhancement is good, yes server rendering is good, but the removal of the AJAX crawling scheme from google bot shouldn't be the motivation.

a-lucas commented 7 years ago

Hello guys, I would like to notify you about ng1-server : https://www.npmjs.com/package/ng1-server

This is my pet project, it uses a headless browser to pre-render any angular 1 web-app.

The main functionalities are :

An angular module that detects the IDLE status, augment the $cacheFactory and tigger preboot.
An advanced caching system, that greatly boost headless rendering by caching every dependecies into redis
The possibility to cache $http calls to
Also once pre-rendered, it is possible to cache the pre-rendered page
And for a last cache bonus, all $http calls on the server are injected into teh client and replayed instantly.
All $logs are server side stored, and can be forwarded to Graylog
I am sure I forgot to mention some other interesting stuff.

I'd really appreciate if you guy have a look at it, and let me know what you think.

Cheers and great new 2017 year for Angular 1 & 2 !

jeffwhelpley commented 7 years ago

Very cool @a-lucas. I like that you have preboot integrated. I don't have a lot of experience with slimer.js, but the challenge with any angular 1.x SSR solution has always been that you either have to:

Emulate the browser on the server side (you are using slimer.js to do this which seems similar to phantom, but another option would be something like Angular.js Server which has built their own custom phantom-like shim.
Heavily restrict Angular 1.x template syntax and force custom APIs everywhere so you can properly abstract out client/server rendering (this is the approach I took with Jangular and Pancakes)

Option number 2 is almost certainly going to scale and perform better, but I gave up trying to make it generalized so other people can use it. There is so much custom stuff in there that in retrospect I would have been better off not even attempting to open source it. So, definitely your path is superior as an OSS solution and it looks like you added a lot of great caching options. I also like that users can take an existing app and just add this on top.

a-lucas commented 7 years ago

Hi @jeffwhelpley ,

Thanks for the quick answer. I had a look at angular.js server you mentionned few month ago, and it seems lile they chose JSDOM to emulate a browser environment. I went jsdom initially, but it came as very unstable and has limited functionality. I haven't tested slimer performances yet, but it seems lile a better alternative than phantomjs. It uses the latest gecko engine which supports ES6 almost completly, and the same api as phantomjs. I though of going the electron way too, but I couldn't find a way to capture all http requests.

Concerning scaling, I intentionally separated the 3 servers , making it easier to scale it in the future (if needed). A good scaling strategy would be imo to modify the bridge_pool so it can span slimer process on severall servers.

As of preboot, I haven't tested it, I am not sure if document.body is the correct way of setting the Approot. I will get this tested soon.

a-lucas commented 7 years ago

@jeffwhelpley pancakes seems lile a very cool concept. I am an absolute fan of code generators.

verishal commented 7 years ago

I am using angular js and phantomjs driver to crawl google careers - https://example.com I tried in the service_args = ['--ignore-ssl-errors=yes','--ignore-ssl-errors=true','--ssl-protocl=any'] Phantomjs version 2.1.1. I tried in the service_args ,but is same ssl error. google crawl not accept the https://example.com/?_escaped_fragment_= request,In https but it still working in http.Like http://example.com/?_escaped_fragment_= .i don't understand why not take request https ? please help me...

when i request the service_args:

phantomjs --disk-cache=no  --ignore-ssl-errors=yes --ignore-ssl-errors=true --ssl-protocol=any /opt/bitnami/apache-tomcat/stt/ROOT/js/angular-seo.js 9090 https://example.com/

Narretz commented 6 years ago

Dedicated server-side rendering support is out of scope at this point of AngularJS development.

verishal commented 6 years ago

Issue has fixed. Problem was in port and SSL. thanks

angular / angular.js

Server side rendering discussion #2104