search

hellock / icrawler

A multi-thread crawler framework with many builtin image crawlers provided.
http://icrawler.readthedocs.io/en/latest/
MIT License
854 stars 174 forks source link

downloader-04 is waiting for new download tasks #2

Closed unluckydan closed 8 years ago

unluckydan commented 8 years ago

I really like your crawler! However, I don't know what's kind of problem about it. Maybe you can add a time constraint for feeder or something else.

hellock commented 8 years ago

Glad that you like it, are you using built-in crawlers? Could you provide some more info or logs about the problem? In fact, when making requests or getting tasks from the queue, the argument timeout is usually set, maybe you are not using the lastest version?

unluckydan commented 8 years ago

I am sorry to trouble you. Maybe I just used pip to install your toolkit so the version is too old. The latest version is working very well thanks for your contribution for my work.^^

2016-04-20 0:46 GMT+08:00 Kai Chen notifications@github.com:

Glad that you like it, are you using built-in crawlers? Could you provide some more info or logs about the problem? In fact, when making requests or getting tasks from the queue, the argument timeout is usually set, maybe you are not using the lastest version?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/hellock/icrawler/issues/2#issuecomment-212011577

Hao Zhu NeuroImage Computation Lab Institute of Cognitive Neuroscience & Learning Beijing Normal University

unluckydan commented 8 years ago

it is still waiting for downloader......... Maybe because I am in china and the great fireware let it down? I have no idea about it.

2016-04-20 11:38 GMT+08:00 unluckydan.bnu unluckydan.bnu@gmail.com:

I am sorry to trouble you. Maybe I just used pip to install your toolkit so the version is too old. The latest version is working very well thanks for your contribution for my work.^^

2016-04-20 0:46 GMT+08:00 Kai Chen notifications@github.com:

Glad that you like it, are you using built-in crawlers? Could you provide some more info or logs about the problem? In fact, when making requests or getting tasks from the queue, the argument timeout is usually set, maybe you are not using the lastest version?

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/hellock/icrawler/issues/2#issuecomment-212011577

Hao Zhu NeuroImage Computation Lab Institute of Cognitive Neuroscience & Learning Beijing Normal University

Hao Zhu NeuroImage Computation Lab Institute of Cognitive Neuroscience & Learning Beijing Normal University

hellock commented 8 years ago

What crawler are you using and which site are you crawling, maybe I can figure it out with more info.

unluckydan commented 8 years ago

just bing. I want to crawl photos from bing, I gave the name, and got photos from your crawls (e.g. jennifer lawrence and someone like this)

2016-04-20 18:54 GMT+08:00 Kai Chen notifications@github.com:

What crawler are you using and which site are you crawling, maybe I can figure it out with more info.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/hellock/icrawler/issues/2#issuecomment-212379055

Hao Zhu NeuroImage Computation Lab Institute of Cognitive Neuroscience & Learning Beijing Normal University

hellock commented 8 years ago

I found the cause of this problem and fixed the bug. Thanks for your information.