Closed summercms closed 3 years ago
The user agent for Gocrawl is as follows:
Gocrawl
Googlebot (gocrawl v0.4)
From reading the closed issue found here: https://github.com/PuerkitoBio/gocrawl/issues/41
The author has said the following: https://github.com/PuerkitoBio/gocrawl/issues/41#issuecomment-139579686
That package uses the name Googlebot in it's user agent but does not own or have any rights to Google's copyright in using their name!
Googlebot
own
any rights
copyright
This repo has been setup to detect fake bots and that user agent will be flagged now and will fail test runner.
The author even suggests for people to use their newer package called fetchbot which doesn't use the name GoogleBot
fetchbot
GoogleBot
Most firewall's block this spoofing attack.
spoofing
Going to remove support for this third-party library as it is creating incorrect results and will not pass.
The user agent for
Gocrawl
is as follows:From reading the closed issue found here: https://github.com/PuerkitoBio/gocrawl/issues/41
The author has said the following: https://github.com/PuerkitoBio/gocrawl/issues/41#issuecomment-139579686
That package uses the name
Googlebot
in it's user agent but does notown
or haveany rights
to Google'scopyright
in using their name!This repo has been setup to detect fake bots and that user agent will be flagged now and will fail test runner.
The author even suggests for people to use their newer package called
fetchbot
which doesn't use the nameGoogleBot
Most firewall's block this
spoofing
attack.Going to remove support for this third-party library as it is creating incorrect results and will not pass.