Closed gromag closed 7 years ago
@gromag can you remove yandex
since yandex supports the escaped fragment crawling protocol? They shouldn't be checked by their user agent.
Also, it might have changed since last time I checked, but bufferbot didn't send a custom user agent of bufferbot
. Instead, they sent a header of x-bufferbot
in the request. So bufferbot
might not work unless they've changed they way they crawl.
Otherwise, this looks good!
@thoop thanks for the guidance, I did not know about those two, I just merged the old ASP.NET list with the Node.js's and while doing that I noticed that were these two extra in the old list so I just carried them over without asking too many questions! =)
Anyway I removed yandex
and changed bufferbot
to x-bufferbot
.
Cheers
Thanks all.
I have extended the list of CrawlerUserAgents of this ASP.NET middleware using the more comprehensive list found in Node.js's middleware https://github.com/prerender/prerender-node/blob/master/index.js#L40 as a reference. I have left "yandex" and "bufferbot" which are currently missing in the Node.js version.
Also note that in the Node.js list "bingbot" was commented out, Todd @ Prerender.io explained how Google, Bing, and Yandex follow the escaped fragment crawling protocol and recommends to add in the
<head>
:<meta name="fragment" content="!">
as per https://developers.google.com/webmasters/ajax-crawling/docs/specification which albeit Google now deprecates it, it is still fully supported.