-
We don't have a robots.txt file yet. I suspect that's why we see things like, e.g., this in /var/log/apache2/ttm_error.log:
```
[Fri Jun 27 20:26:14 2014] [error] [client 75.168.181.198] File does n…
-
Empty disallow statements `Disallow:` are interpreted as disallow all while they should be interpreted as allow all.
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Examples
-
5th one down. Neither staging nor test should appear in Google
![staging](https://cloud.githubusercontent.com/assets/5955396/6593793/cf415bdc-c7db-11e4-8324-7f3b6719c3b7.jpg)
-
As discussed, https:// sites that have multiple authors may be surprised to discover that user a can now overwrite content for user b.
What's needed:
1) text (Security Considerations?) explaining t…
-
Way too many "mobile" versions of applications are dumbed-down and nearly useless. Often there is so little screen left between big sloppy borders that it's like looking at the world through a mail s…
-
Hi,
Does node-crawler obey the robots.txt exclusion standard, described at http://www.robotstxt.org/wc/exclusion.html#robotstxt and
robots META tag, as described at http://www.robotstxt.org/wc/meta-u…
-
Converted from [SourceForge issue 1323649](http://sourceforge.net/support/tracker.php?aid=1323649), submitted by javahollic
I'd like to use linkchecker against a sharepoint based
website, it currentl…
-
Add a robots.txt with a stanza to allow indexing, with a delay. cc @kltm @harryhoch
-
Hi,
Nice class. I doesn't recognise the directive for `Crawl-Delay` though. (See: http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive and https://ahrefs.com/robot/)
-
Mget should download and respect /robots.txt "Robot Exclusion Standard" and ``.
More information here:
http://en.wikipedia.org/wiki/Robots_exclusion_standard
Thanks to the multithreading nature of Mg…