-
Ubuntu 18.04 Java 11 HDD
Crawler Queue Size was good at 10000 instead of 200 it reduces DNS load.
[https://twitter.com/smokingwheels/status/1577306387960696845](https://twitter.com/smokingwheels/s…
-
Can a feature/flag be added to allow for crawling sites that need credentials for accessing specific pages?
-
Search doesn't poll the internet. Nothing good there.
Doesn't communicate with other known archives or databases of interactive fiction e.g. ifarchive, itchio... IFDB is all we need to find the goo…
-
@katehausladen provided some initial analysis accuracy analysis as shown in [our draft paper](https://drive.google.com/open?id=1lkE6BdyVFfmE2fdPvDWKkvLcr6Rk8wqV&usp=drive_fs) (section 3.5). Starting w…
-
Currently don't see any SSL support for crawling sites with SSL enabled?
-
### Describe your issue
See the screenshot below. My issue is that I would like to login to this service, and some other services having same issues.
How would this be possible with the current cod…
-
```[tasklist]
### Tasks
- [x] Review existing research
- [x] Conduct new research if needed
- [x] [Draft standard in Google docs for internal sharing](https://docs.google.com/document/d/1mdRTyrlPZoCsj…
-
-
In addition to Scanner's static analysis, Scanner should support dynamic analysis of sites.
This would consist of finding site urls and crawling them, while recording what happens, such as expensive …
-
It would be good if there was a way to set cookies for requests to allow for crawling sites that require authentication.
Is there currently a way to do this, or is this feature planned?