-
- Site: [https://localhost:5000](https://localhost:5000)
- Site: [http://localhost:5000](http://localhost:5000)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set** [10038] total:…
-
But see this: https://developers.google.com/search/docs/crawling-indexing/block-indexing
-
насроить robots.txt так что бы на сендах по ПРкам запрещалось роботам посещать страницы а на релизных сендах разрешалось
еще сделать sitemap.xml
убрать лишние стенды и настроить от них редирект …
-
currently js-crawler does not look and follow robots.txt file. This feature should be added. There should be one variable to check if crawler reads and follow robots.txt or nor.
-
Just a note that if you are publishing your crawler results to please respect sites which host a robots.txt restricting spidering.
For instance: gopher://gopher.black/0/robots.txt
Thanks
-
- Site: [https://localhost:5000](https://localhost:5000)
- Site: [http://localhost:5000](http://localhost:5000)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set** [10038] total:…
-
- Site: [https://nr-bcmi-test-frontend.apps.silver.devops.gov.bc.ca](https://nr-bcmi-test-frontend.apps.silver.devops.gov.bc.ca)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set**…
-
- Site: [https://localhost:5000](https://localhost:5000)
- Site: [http://localhost:5000](http://localhost:5000)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set** [10038] total:…
-
- Site: [https://localhost:5000](https://localhost:5000)
- Site: [http://localhost:5000](http://localhost:5000)
**New Alerts**
- **SQL Injection** [40018] total: 1:
- [http://localhost:500…
-
- Site: [https://localhost:5000](https://localhost:5000)
- Site: [http://localhost:5000](http://localhost:5000)
**New Alerts**
- **Content Security Policy (CSP) Header Not Set** [10038] total:…