-
### Description of the bug
At first I tried it on a local News site and got blocked by Cloudflare. So I thought I'd use a Medium article and got the same blocked by Cloudflare page.
[Link 1](http…
-
* add translation code
* add texts translated to German
* localize number and date formats
* show errors if input fields have invalid format
-
Adding an ad-blocker seems to make crawling much easier. On two sites I've tested, without an adblocker the number of requests is an order of magnitude higher than with it, and on one of the two sites…
-
e.g. for the same person's accounts on each silo. could use indieauth to also let them attach their domain, but not strictly necessary.
-
URL you wish to be added:
vidyo.us.to
hubnsfw.com
Why you believe this should be added:
porn sites not yet blocked, discovered by crawling reddit
Add to list:
porn
Other info you think we…
-
### Before submitting your bug report
- [ ] I believe this is a bug. I'll try to join the [Continue Discord](https://discord.gg/NWtdYexhMs) for questions
- [ ] I'm not able to find an [open issue](ht…
-
Sometimes I get filtered as a spam bot if I'm crawling a site, and this is sent to a central service which forbids my ID from many sites.
I think we could solve this by slowing down the call rate. …
-
Hi, I've been testing out the CLI version of the tool and am absolutely loving it so far. It's a very solidly put together tool with Great performance and information output.
I have been trying to…
-
Hi,
On some sites, the crawler hangs up by throwing the following error:
```
Error: Couldn't unzip robots.txt response body
0|www | at decodeAndReturnResponse (/var/www/node_modules/…
-
Description:
Enhance the existing web crawler to support crawling and extracting content from websites that rely heavily on JavaScript for rendering their content. This feature will involve integra…