-
Hello,
I have noticed that most instances keep the default robots.txt file, which allows any page to be indexed by search engines. As privacy is a concern for many users in the fediverse, I believe i…
symac updated
7 years ago
-
robots `host` directive is not recognized.
> User-agent: *
> Disallow: /search/
> Disallow: /goto?
> Disallow: /login
> Disallow: /api
> Host: https://site.com
> Sitema…
-
While conceptually I applaud Calypso as a huge advance for WordPress, as far as underlying technological choices, one big reason I chose [Mithril](https://lhorie.github.io/mithril/mithril.html) over R…
-
All we have is a bunch of outdated information that nobody's able to fix now, and it's worse than before. Lots of that information doesn't make sense to put in a manual. If people find outdated or bro…
-
I'm just thinking if the configs are safe for production use as is.
-
Look into methods to prevent search bots indexing the staging site.
A good initial solution would be to add the robots.txt Host directive.
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Host…
-
No scrapin' for this please!
-
Because our site navigation cannot be crawled so easily with a non-javascript enabled web crawler since https://github.com/loklak/loklak_server/issues/394, we need a sitemap which links all available …
-
Hello.
Please, if possible, add the Host field.
SEO specialists is ask.
```
User-agent: *
Disallow: /admin/
+ --> Host: example.com
Sitemap: http://example.com/sitemap.xml
```
-
The following robots.txt will be interpreted by the class as denying everything:
User-Agent: *
Disallow: /*
Allow: /$
However, Google will interpret this (correctly) as allowing the homepage only.
…