Running this with google image search seems to not work anymore. Bing seems to work fine.
$ python icrawlertest.py
2020-09-04 09:07:03,115 - INFO - icrawler.crawler - start crawling...
2020-09-04 09:07:03,116 - INFO - icrawler.crawler - starting 1 feeder threads...
2020-09-04 09:07:03,120 - INFO - feeder - thread feeder-001 exit
2020-09-04 09:07:03,120 - INFO - icrawler.crawler - starting 1 parser threads...
2020-09-04 09:07:03,123 - INFO - icrawler.crawler - starting 4 downloader threads...
2020-09-04 09:07:03,829 - INFO - parser - parsing result page https://www.google.com/search?q=cat&ijn=0&start=0&tbs=isz%3Al%2Cic%3Aspecific%2Cisc%3Aorange%2Csur%3Afmc%2Ccdr%3A1%2Ccd_min%3A01%2F01%2F2017%2Ccd_max%3A11%2F30%2F2017&tbm=isch
Exception in thread parser-001:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/home/yc/.local/lib/python3.8/site-packages/icrawler/parser.py", line 104, in worker_exec
for task in self.parse(response, **kwargs):
File "/home/yc/.local/lib/python3.8/site-packages/icrawler/builtin/google.py", line 157, in parse
meta = json.loads(txt)
File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/local/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2020-09-04 09:07:08,146 - INFO - downloader - no more download task for thread downloader-002
2020-09-04 09:07:08,147 - INFO - downloader - thread downloader-002 exit
2020-09-04 09:07:08,148 - INFO - downloader - no more download task for thread downloader-001
2020-09-04 09:07:08,149 - INFO - downloader - no more download task for thread downloader-003
2020-09-04 09:07:08,149 - INFO - downloader - thread downloader-001 exit
pasqLisena
commented
4 years ago
Running this with google image search seems to not work anymore. Bing seems to work fine.