-
Hi,
Thanks for this wonderful project.
What is the recommended way to add rate limiting to the spider?
I normally add a randomized delay like for scraping:
```python
requests.get(url)
ti…
-
-
## 项目推荐
- 项目名称:Ruia
- 项目地址:https://github.com/howie6879/ruia
- 项目后续更新计划:
- 插件开发
- 维护
- 项目描述:
Python轻量异步爬虫框架,An async web scraping micro-framework based on asyncio.
- 推荐理由:
…
-
What are the possible options for scraping multiple websites e.g. through a list or a file and saving the results in a database?
-
Mainly because the server rejected our request.
```python
import asyncio
from ruia import Item, TextField, AttrField
class HackerNewsItem(Item):
target_item = TextField(css_select='tr.a…