-
**Description**
After activating button in the dialog which closes it, navigation should start from the element which opened the dialog not from the first element on the webpage. Otherwise users, espe…
-
### Pitch
In August, GPTbot block was merged into the code https://github.com/mastodon/mastodon/pull/26396. Now, Google has a robots.txt policy for Bard and future Google AI models with user agent Go…
-
### Feature Request: Enhance Static Website with Anti-Crawling, Encrypted Blog Access, and Simplified Theme Customization
#### 1. Anti-Crawling Mechanisms
- Add a `robots.txt` file to control acce…
-
All the left nav links from the Docs pages are missing `/docs` in the file path, which means Botify (and potentially other crawlers) interprets them as a 404, even though the page displays fine when t…
-
# Feature Request
Create a sample module that guides developers through the implementation of Federated Catalog (FC) extensions.
## Which Areas Would Be Affected?
A new `federated_catalog` mo…
-
I checked the source view of the live demo page and there is no content there.
The crawlers only see what you can get from the source.
Twois updated
6 months ago
-
Re implement scrapers from the upstream repo
- 1337x
- torrent9
- nyaaPantsu
- nyaaSis
- eztv
-
List of broken crawlers
-
### What version of Bun is running?
1.1.22
### What platform is your computer?
Linux 6.5.0-45-generic x86_64 x86_64
### What steps can reproduce the bug?
Hello Bun Community,
I'm using [apify/…
-
As a builder I sometimes want to edit the robots.txt file. Original discussion https://discord.com/channels/955905230107738152/1292740898832257055
One of the use cases mentioned was: blocking AI cr…