yjg30737
commented
1 year ago don't know why that happened.. for me it works.
sure you installed required packages with running "pip install -r requirements.txt" :)
maybe you i think you can ask this to icrawler repo directly
Unfortunately the code doesn't work for me. When I run the program I get this this message:
">>
started
2023-09-03 20:31:45,500 - INFO - icrawler.crawler - start crawling...
2023-09-03 20:31:45,509 - INFO - icrawler.crawler - starting 1 feeder threads...
2023-09-03 20:31:45,517 - INFO - feeder - thread feeder-001 exit 2023-09-03 20:31:45,517 - INFO - icrawler.crawler - starting 2 parser threads... 2023-09-03 20:31:45,531 - INFO - icrawler.crawler - starting 4 downloader threads... C:\Users\user\AppData\Roaming\Python\Python311\site-packages\urllib3\connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'www.google.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( C:\Users\user\AppData\Roaming\Python\Python311\site-packages\urllib3\connectionpool.py:1095: InsecureRequestWarning: Unverified HTTPS request is being made to host 'consent.google.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings warnings.warn( 2023-09-03 20:31:46,425 - INFO - parser - parsing result page https://www.google.com/search?q=10&ijn=0&start=0&tbs=ic%3Acolor&tbm=isch Exception in thread parser-001: Traceback (most recent call last): File "C:\Program Files\Python311\Lib\threading.py", line 1038, in _bootstrap_inner self.run() File "C:\Program Files\Python311\Lib\threading.py", line 975, in run self._target(*self._args, self._kwargs) File "C:\Users\user\AppData\Roaming\Python\Python311\site-packages\icrawler\parser.py", line 94, in worker_exec for task in self.parse(response, kwargs): TypeError: 'NoneType' object is not iterable 2023-09-03 20:31:47,540 - INFO - parser - no more page urls for thread parser-002 to parse 2023-09-03 20:31:47,540 - INFO - parser - thread parser-002 exit 2023-09-03 20:31:50,536 - INFO - downloader - no more download task for thread downloader-003 2023-09-03 20:31:50,537 - INFO - downloader - no more download task for thread downloader-004 2023-09-03 20:31:50,538 - INFO - downloader - no more download task for thread downloader-001 2023-09-03 20:31:50,539 - INFO - downloader - no more download task for thread downloader-002 2023-09-03 20:31:50,539 - INFO - downloader - thread downloader-003 exit 2023-09-03 20:31:50,540 - INFO - downloader - thread downloader-004 exit 2023-09-03 20:31:50,541 - INFO - downloader - thread downloader-001 exit 2023-09-03 20:31:50,542 - INFO - downloader - thread downloader-002 exit 2023-09-03 20:31:51,542 - INFO - icrawler.crawler - Crawling task done! finished"
Can someone help?