Help web crawlers look at pages

[wikett]

Hi there,

I am trying to improve the SEO for my website following this article: http://www.michaelbromley.co.uk/blog/171/enable-rich-social-sharing-in-your-angularjs-app

Does someone know how can I use .htaccess in heroku? I have changed the .htaccess file from \dist\public folder adding this:

<ifModule mod_rewrite.c>
RewriteEngine On

# allow social media crawlers to work by redirecting them to a server-rendered static version on the page
RewriteCond %{HTTP_USER_AGENT} (facebookexternalhit/[0-9]|Twitterbot|Pinterest|Google.*snippet)
RewriteRule album/(\d*)$ http://www.otherdomain.comr/static-page.php?id=$1 [P]

</ifModule>

but doesn't work.

Any suggestion?

Thanks in advance

[kingcody]

I could be wrong, since I've never used heroku, but I don't think that is going to work. A .htaccess file is typically used to configure an apache web server. This project is using nodejs as a web server, which does not parse/respect .htaccess config files.

Most likely you will need to use a rewrite method available to nodejs.

[Awk34]

I have been looking into improving support for crawlers. The main problem with any SPA is that they require JS to render any page, and most web crawlers will not run any JS code (you can see what this looks like if you tell your browser to not run any JS code)

[Svenskunganka]

Prerender uses PhantomJS to cache a SPA and executes JS. I've been using it for this very generator for a long time and it works really well.

[Awk34]

Good to know @Svenskunganka . wink PRs welcome wink

[Svenskunganka]

@Awk34 I'll see what I can do! Never worked with Yeoman generators before, but I do love challenges! ;D

[kingcody]

@Svenskunganka, that could be really nice. I look forward to seeing what you come up with.

[Svenskunganka]

@kingcody The issue is that Prerender has to be a standalone server, so it can't be "embedded" into the main app. On deployment, you have to either have the prerender service running or signed up for an account with prerender.io unfortunately. Perhaps I could build a middleware for Express and not rely on Prerender at all.

Any input on this?

[kingcody]

@Svenskunganka I don't see why you coudn't 'embedded' it by including it as a dependency and then simply require('prerender/server'). You would need to have already set process.env.PRERENDER_SERVICE_URL, but that could easily be taken care of with configs. I've never used prerender but do you see any problem with this method?

[Svenskunganka]

@kingcody I've been looking into this a bit and Prerender doesn't seem that modularily built. Can't really pass a http server to the prerender/server object, so we'd have to wrap that. Nothing that can't be done but would probably be easier and cleaner to write an Express middleware for it instead.

I'll dig into it a bit more and report back what seems like the best choice.

[wikett]

Ok then my idea it is to create a middleware to redirect the calls from Facebook,Twitter, etc. But I don´t know where I have to write the middleware (in express.js?????) in \server\app.js??? or \server\routes.js??

I write a pseudo-code of my idea if(user-agent === "FACEBOOK") redirect("www.otherdomain.com/static-page.php?id="+req.params.id)

It make sense?

[Awk34]

Yeah, that's the idea. If we detect that the user is a web crawler, we want to serve them static content with no JS instead of the regular app.

[Awk34]

Ideally though we could do this all on our servers without having to contact a separate service for pre-rendering

[glenn-coessens]

http://www.ng-newsletter.com/posts/serious-angular-seo.html This link could be interesting (routing) in combination with a PhantomJS module or some other header-less browser which can execute javascript and serve the result back to the node server.

[riddla]

http://searchengineland.com/tested-googlebot-crawls-javascript-heres-learned-220157?utm_source=javascriptweekly&utm_medium=email might also be interesting. Google was and is actually quite good at parsing JavaScript content.

An example from my experience: the string "Geben Sie den angezeigten Text ein" within the newsletter subscription on http://www.radio.de/ is inserted via Angular translations within a JavaScript template. Furthermore it is only revealed/inserted if the user starts to enter within the email address field:

Nevertheless the google index has it without problems:

(https://www.google.de/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=site:http:%2F%2Fwww.radio.de%2F+%22geben+sie+den+angezeigten+text+ein%22)

[wikett]

Hi, I have my own project using generator-angular-fullstack and I dont have problem with Google. The issue is when you try to share one link on Facebook. My idea is to create a middleware in express.js to know if the user-agent is Facebook and redirect to a html page generated in node.js. But... I am not sure how to do it in generator-angular-fullstack. I am newbie in express.js and node.

My approach is in \server\thing\index.js add the middleware (when you call to 'thing' url): \server\thing\index.js

'use strict';

var express = require('express');
var controller = require('./thing.controller');

var router = express.Router();

router.use(function (req, res, next) {
   // check user-agent (Facebook)
    res.sendFile('client/index-fb.html');
});

router.get('/', controller.index);
router.get('/:id', controller.show);
router.post('/', controller.create);
router.put('/:id', controller.update);
router.patch('/:id', controller.update);
router.delete('/:id', controller.destroy);

module.exports = router;

But it doesn't work for me.

Does anyone know how create a middleware to create a .html underd demand??

Thanks in advance

[glenn-coessens]

Never written a middleware component before but i think it should look something like this:

app.get('/', function (req, res, next) { if (user agent is facebook) next('route'); // else pass the control to the next middleware in this stack else next(); // }, function (req, res, next) { // render a regular page res.send(regular); });

// handler for / which renders a special page app.get('/', function (req, res, next) { //call function with callback to generateStaticHTML GenerateStaticHTML(req,res, function(result){ //you will need the 'req' and 'res' parameters to generate a page res.send(result); }); }); GenerateStaticHTML(req,res, callback){ //generate a page with PhantomJS or something else and put the result into a 'result' variable, //use callback(result) inside this function to get back to the function above };

Do the same for GET '/:id'

Again i haven't tested this and you will probably want to seperate the 'phantomJS logic' into another file.