-
BPO | [43597](https://bugs.python.org/issue43597)
--- | :---
Nosy | @berkerpeksag, @Tchinmai7
PRs | python/cpython#24984python/cpython#24986
*Note: these values reflect the state of the issue at the …
-
This crashes after about a minute:
```
~/.local/bin/wpull3 --concurrent 10 -r --sitemaps http://2ch.en.utf8art.com/
```
```
INFO Fetching ‘http://2ch.en.utf8art.com/’.
INFO Fetched ‘http://2ch.en.ut…
ivan updated
8 months ago
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
Hi, we have recently switched from Reppy to Protego for our robots.txt parser. All seems fine, except we noticed a few differences between Reppy and Protego in the URLs we are crawling - essentially, …
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
Some 🔍 facts / ❤️ feelings / 💡 ideas about this:
- ❤️ I feel proud of our custom shortlink tool, and don't wish to use any other.
- 🔍 Clients/collaborators have expressed enthusiasm for our shortl…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…
-
```
What steps will reproduce the problem?
1. Find a website where robots.txt has something similar to
User-agent: *
Crawl-delay: 80
2. Run the crawler with a parser
What is the expected output? What…