-
This is a regression. Was working as of [2.8](https://github.com/hydroshare/hydroshare/releases/tag/2.8)
h…
-
Hi, I wanted to install the userscript today, but when I wanted to view the lyrics instead of lyrics popping up this showed up:
```
User-agent: *
Allow: /
Disallow: /local/
Disallow: /download/
…
-
## Ability to edit the robots.txt file
The `robots.txt` file is a simple text file placed in the root directory of a website. It serves as a set of instructions for web crawlers (like those used b…
-
Add robots.txt file to improve SEO.
-
Hello!
I was wondering what process you recommend for setting the headers [so Google, etc. doesn't index the robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)…
-
When wget2 is used with --recursive it is always looking for the robots.txt file. This happens even when the file is not present on the server and also when --no-robots is used. A quick reproducer bel…
-
We should to block certain web crawlers from putting extra load on the database. We need to make a list of URLs that should be excluded from crawling and add them to `robots.txt` file.
-
- Site: [https://ipadcheckout.boring9.dev](https://ipadcheckout.boring9.dev)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set** [10038] total: 2:
- [https://ipadcheckout.borin…
-
We are working on improving Google indexing and have noticed that pages in the form /items/*/request-a-copy are indexed in Google. Would it make sense to further enhance robots.txt with the optional s…
-
We make several dictionaries web-accessible for testing and internal demos, which should not be considered as deploy-ready. For this, we should update the robots.txt files for those deployments to en…