-
I have been going through the GSOC projects of last summer, and I was interested in asyncio prototype. Now, there was one issue that came after I searched through the details of asyncio. Asyncio does …
-
按照现在的设计思路，应该是1个网站一个spider的模式了，但是想同时抓取多个网站，有没有办法同时开多个spider，同时抓取。
而且设计成类方法和类变量的话感觉不是很灵活，对于这种情况。
-
when i wrote two simple spider
```
class Bing(Spider):
name = "bing"
def start_requests(self):
for _ in range(1000):
yield Request("http://bing.com", dont_filter…
-
Hi,
I wanted to know if Scrapy had an internal mechanism, using Twisted or other, to make request without callback like this:
```python
import scrapy
from scrapy import spiders, FormRequest
…
-
## Background
Currently, the techniques async version gRPC uses is far behind network model implemented in several languages including cpp.
gRPC uses completion queue to notify users when an eve…
-
Works fine locally, but getting this error when running the spider on scrapinghub. Any ideas?
```
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/twisted/interne…
-
## Short version based on findings
I've been working on the Z2JH projects CI system a lot recently, which often fails seemingly at random. I figured it out I think.
proxy.py's function `api_requ…
-
from ruia import AttrField, TextField, Item
from ruia_pyppeteer import PyppeteerSpider as Spider
from ruia_pyppeteer import PyppeteerRequest as Request
from ruia_ua import middlewar…
-
## 项目推荐
- 项目名称：Ruia
- 项目地址：https://github.com/howie6879/ruia
- 项目后续更新计划：
- 插件开发
- 维护
- 项目描述：
Python轻量异步爬虫框架，An async web scraping micro-framework based on asyncio.
- 推荐理由：
…
-
Hi there!
I hit an issue when trying to use `aiomysql` with the `ProactorEventLoop` on Windows. This is pretty much a show stopper for me given the limitations of the (default) `SelectorEventLoop`…