-
```
What steps will reproduce the problem?
1. put Disallow: /example/ in robots.txt
2. or put /example/ in Excluded URL patterns
3. what it works if you put /example/* in Excluded URL patterns
Wha…
-
```
What steps will reproduce the problem?
1. put Disallow: /example/ in robots.txt
2. or put /example/ in Excluded URL patterns
3. what it works if you put /example/* in Excluded URL patterns
Wha…
-
- Site: [http://172.32.52.95:5058](http://172.32.52.95:5058)
**New Alerts**
- **Storable and Cacheable Content** [10049] total: 3:
- [http://172.32.52.95:5058](http://172.32.52.95:5058)
-…
-
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive
-
### DESCRIPTION
According to the readme, we are supposed to be able to change the contents of 'robots.txt' via either a config option - in 'config.php' - or by using an appropriately-named field in t…
-
```
Robots could be dangerous (now unknown security issues etc.), used to abuse
server by users or just very, very annoying. Those problems will arise on
public servers, used by hundreds people (in …
-
```
What steps will reproduce the problem?
1. Crawl a page with in
between tags
What is the expected output? What do you see instead?
Expected: Outgoing URLs are not listed if content is set to "N…
-
Please could you block test servers from being crawled by google, yahoo, other bots?
https://github.com/akvo/akvo-web/issues/577
-
_Original author: inedi...@gmail.com (October 26, 2012 09:03:06)_
What steps will reproduce the problem?
1. Create a Virtual Server with the option vServers > New > CMS>Wordpress
2. Configur…
-
```
According to this site, Google supports a "noindex: xxx" directive:
http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex
-your-site/
```
Original issue reported on c…