search

greengerong / Prerender_asp_mvc

Prerende client for asp.net mvc
http://prerender.io/
86 stars 39 forks source link

Extendend list of CrawlerUserAgents #26

Closed gromag closed 7 years ago

gromag commented 7 years ago

I have extended the list of CrawlerUserAgents of this ASP.NET middleware using the more comprehensive list found in Node.js's middleware https://github.com/prerender/prerender-node/blob/master/index.js#L40 as a reference. I have left "yandex" and "bufferbot" which are currently missing in the Node.js version.

Also note that in the Node.js list "bingbot" was commented out, Todd @ Prerender.io explained how Google, Bing, and Yandex follow the escaped fragment crawling protocol and recommends to add in the <head> :

<meta name="fragment" content="!">

as per https://developers.google.com/webmasters/ajax-crawling/docs/specification which albeit Google now deprecates it, it is still fully supported.

thoop commented 7 years ago

@gromag can you remove yandex since yandex supports the escaped fragment crawling protocol? They shouldn't be checked by their user agent.

Also, it might have changed since last time I checked, but bufferbot didn't send a custom user agent of bufferbot. Instead, they sent a header of x-bufferbot in the request. So bufferbot might not work unless they've changed they way they crawl.

Otherwise, this looks good!

gromag commented 7 years ago

@thoop thanks for the guidance, I did not know about those two, I just merged the old ASP.NET list with the Node.js's and while doing that I noticed that were these two extra in the old list so I just carried them over without asking too many questions! =) Anyway I removed yandex and changed bufferbot to x-bufferbot.

Cheers

greengerong commented 7 years ago

Thanks all.