Spiderable returns empty body

[raix]

I'v added the spiderable package, it kinda renders - used curl to check but body is empty?

curl meteorrain.meteor.com

Really just needed a img tag and some text for twitter/fb to use.

EDIT: At first I tested with fb - It didn't find any img tags or text. I've read the spiderable code - can see that only facebook bot is supported and there might be an issue with node?

Knowing the spiderable is a temperary approach - Should I try fixing or try getting templates rendered serverside? (guess it would require some input from core, eg. strategy for js/non js bots - use robots.txt or only support major bots)

[svasva]

Spiderable only works for requests that match AJAX Crawl Specification ( https://developers.google.com/webmasters/ajax-crawling/ ). Try doing curl http://meteorrain.meteor.com/?_escaped_fragment_=

[raix]

Thanks @erundook, I've read the google spec - So spiderable is only usefull for ajax enabled crawlers actually setting the _escaped_fragment_. Guess it doesn't solve facebook&linkedin etc.

Just for the fun of it I tried setting a fictive parametre in fb link eg. meteorrain.meteor.com?hash= and this works in fb, but bit ugly compared with a simple meteorrain.meteor.com - wich doesn't work i fb/linkedin.

Would it be possible to check the agents and eg. set the ?_escaped_fragment_= for these if they don't do it them selfs? -before passing on to phantom.js (I'll have a look at spiderable.js I think)

[svasva]

@raix you could do that (useragent detection) with some kind of reverse proxy (nginx for exapmle).

[raix]

@erundook : nope, dont have to use proxy, I just have to make the spiderable.js work... It has regular ex. that tests agent for 'facebookexternalhit', I'm gonna figure out why it seems not to work, and add some test for linkedin too. but thanks for your suggestion.

[raix]

@erundook: Guess facebook's user agent changed from lower case to CamelCase - the spiderable only tested lower case. Spiderable only supported the fb bot.

I've fixed the test and added the linkedin bot.

[raix]

@erundook: ... and thanks for your input :)

[glasser]

Pushed @raix's patch.

[svasva]

It would be awesome if this would be configurable on per-app basis.

[raix]

Just to clearify, Like eg.: Meteor.spiderableAgents = [/^onlyBotAllowed/]?

[svasva]

Yes, just like that, but with a way to add agents instead of overriding the whole array. Like Meteor.spiderableAgents.push('someBot')

On Fri, Mar 1, 2013 at 5:25 PM, Morten N.O. Nørgaard Henriksen < notifications@github.com> wrote:

Just to clearify, Like eg.: Meteor.spiderableAgents = [/^onlyBotAllowed/]?

— Reply to this email directly or view it on GitHubhttps://github.com/meteor/meteor/issues/721#issuecomment-14289038 .