-
I have included several websites for testing, but no matter what questions are asked, the answers I receive generally mean that there is no relevant content context
![image](https://github.com/user-a…
-
Maybe add other sites in the future/Rewrite crawling code to be more flexible
araml updated
4 years ago
-
Can a feature/flag be added to allow for crawling sites that need credentials for accessing specific pages?
-
Right now, all GtS instances serve a simple hardcoded `robots.txt` that disallows all crawling:
```
User-agent: *
Disallow: /
```
The code for this is here: https://github.com/superseriousbus…
-
Ubuntu 18.04 Java 11 HDD
Crawler Queue Size was good at 10000 instead of 200 it reduces DNS load.
[https://twitter.com/smokingwheels/status/1577306387960696845](https://twitter.com/smokingwheels/s…
-
I am opening this issue because https://github.com/dgtlmoon/changedetection.io/discussions/1979 was deleted apparently without a final state (rejected/accepted).
Obersvation: Pure changedetection (…
-
Currently don't see any SSL support for crawling sites with SSL enabled?
-
In addition to Scanner's static analysis, Scanner should support dynamic analysis of sites.
This would consist of finding site urls and crawling them, while recording what happens, such as expensive …
-
### Describe your issue
See the screenshot below. My issue is that I would like to login to this service, and some other services having same issues.
How would this be possible with the current cod…
-
It would be good if there was a way to set cookies for requests to allow for crawling sites that require authentication.
Is there currently a way to do this, or is this feature planned?