search

elliotgao2 / gain

Web crawling framework based on asyncio.
GNU General Public License v3.0
2.03k stars 207 forks source link

Add hooks before download and after download. #44

Open songww opened 6 years ago

songww commented 6 years ago

我们想在下载前或者下载后加一些东西，比如保存整个url。或者针对不同的状态码做不同的处理，比如3xx的重定向，5xx时重试等。

kwuite commented 6 years ago

Google translate from @songww:

We want to add something before or after the download, such as saving the entire url. Or do different processing for different status codes, such as 3xx redirection, 5xx retry, etc.

kwuite commented 5 years ago

@songww ,

Can you elaborate a bit more on your before and afterhooks. I might be able to implement this for you if you specify a bit more in detail why you need this and how you would use it.

songww commented 5 years ago

Just like middleware of scrapy.

@kwuite