-
"robots", stores the sent Robots-X-Tag HTTP headers. Is there a way to simply remove the spaces from these? Right now it shows `noindex, follow` and `noindex,follow` as different entities.
See scree…
-
```
$Subject
```
Original issue reported on code.google.com by `a.c.neumann` on 18 Jul 2010 at 7:44
-
```
$Subject
```
Original issue reported on code.google.com by `a.c.neumann` on 18 Jul 2010 at 7:44
-
```
$Subject
```
Original issue reported on code.google.com by `a.c.neumann` on 18 Jul 2010 at 7:44
-
Hi all,
More than an issue it should be read as a feature request.
I was wondering whether it is possible or not to optionally ignore the robots exclusion protocol so you can check for broken links…
-
Hi @johtani, I have faced with unexpected behavior when I try to grub site with an enabled robots_txt option.
robots.txt like:
User-agent: *
Disallow: /
and my expected result that the site w…
-
```
$Subject
```
Original issue reported on code.google.com by `a.c.neumann` on 18 Jul 2010 at 7:44
-
Bug we found today (Friday) where the robot will stop responding to the ball completely after some time. Seems not to be a crash, but timing related (e.g. a hang somehow). Teensy LED is not blinking, …
-
We're currently keeping bots away from certain pages (namely search results and person record pages) with a meta tag, implemented [here](https://github.com/google/personfinder/blob/f2b07e84d6dcbecef7f…
-
```
$Subject
```
Original issue reported on code.google.com by `a.c.neumann` on 18 Jul 2010 at 7:44