2022-02-16 14:06:22 [scrapy.core.scraper] ERROR: Spider error processing <GET https://s.weibo.com/weibo?q=%E5%8F%8C%E5%87%8F&scope=ori&suball=1×cope=custom:2021-07-24-0:2022-02-12-0&display=0&retcode=6102> (referer: None)
Traceback (most recent call last):
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/defer.py", line 120, in iter_errback
yield next(it)
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 353, in next
return next(self.data)
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 353, in next
return next(self.data)
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output
for x in result:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 342, in
return (_set_referer(r) for r in result or ())
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 40, in
return (r for r in result or () if _filter(r))
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in
return (r for r in result or () if _filter(r))
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/szhang/scripy/weibo-search/weibo/spiders/search.py", line 107, in parse
for weibo in self.parse_weibo(response):
File "/home/vsi/szhang/scripy/weibo-search/weibo/spiders/search.py", line 420, in parse_weibo
comments_count = re.findall(r'\d+.*', comments_count)
File "/usr/lib/python3.7/re.py", line 223, in findall
return _compile(pattern, flags).findall(string)
TypeError: expected string or bytes-like object
如果把Line 420注释掉,仅能抓取几条数据。
请帮忙查看,谢谢!
2022-02-16 14:06:22 [scrapy.core.scraper] ERROR: Spider error processing <GET https://s.weibo.com/weibo?q=%E5%8F%8C%E5%87%8F&scope=ori&suball=1×cope=custom:2021-07-24-0:2022-02-12-0&display=0&retcode=6102> (referer: None) Traceback (most recent call last): File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/defer.py", line 120, in iter_errback yield next(it) File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 353, in next return next(self.data) File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/utils/python.py", line 353, in next return next(self.data) File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable for r in iterable: File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/offsite.py", line 29, in process_spider_output for x in result: File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable for r in iterable: File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/referer.py", line 342, in
return (_set_referer(r) for r in result or ())
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/urllength.py", line 40, in
return (r for r in result or () if _filter(r))
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/spidermiddlewares/depth.py", line 58, in
return (r for r in result or () if _filter(r))
File "/home/vsi/.local/lib/python3.7/site-packages/scrapy/core/spidermw.py", line 56, in _evaluate_iterable
for r in iterable:
File "/home/vsi/szhang/scripy/weibo-search/weibo/spiders/search.py", line 107, in parse
for weibo in self.parse_weibo(response):
File "/home/vsi/szhang/scripy/weibo-search/weibo/spiders/search.py", line 420, in parse_weibo
comments_count = re.findall(r'\d+.*', comments_count)
File "/usr/lib/python3.7/re.py", line 223, in findall
return _compile(pattern, flags).findall(string)
TypeError: expected string or bytes-like object
如果把Line 420注释掉,仅能抓取几条数据。
请帮忙查看,谢谢!