Seems Google tends to create more and more robots with names.
Also looking at other existing headers. Google ads urls to their bots
+http://www.google.com/adsbot.html or +https://support.google.com/webmasters/answer/1061943
Maybe it would be a good idea to regex for google domain url?
Hello,
I did not see any rule that would detect following:
AdsBot-Google (+http://www.google.com/adsbot.html)Google-Adwords-Instant (+http://www.google.com/adsbot.html)Seems Google tends to create more and more robots with names.
Also looking at other existing headers. Google ads urls to their bots
+http://www.google.com/adsbot.htmlor+https://support.google.com/webmasters/answer/1061943Maybe it would be a good idea to regex for google domain url?