Lodour / Weibo-Album-Crawler

A multiprocessing crawler for weibo albums.
MIT License
99 stars 34 forks source link

大图数据无效 #7

Closed Lodour closed 6 years ago

Lodour commented 6 years ago

当访问频率过快时,所获取的大图数据是无效的,导致以下报错。

Traceback (most recent call last):
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/core.py", line 74, in __download_album
    result, path = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/_base.py", line 398, in result
    return self.__get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/concurrent/futures/thread.py", line 55, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/core.py", line 122, in __download_pic
    path = os.path.join(path, self.__make_photo_name(pic))
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/core.py", line 148, in __make_photo_name
    f, p = large.get('feed_id'), large['pic_name']
AttributeError: 'NoneType' object has no attribute 'get'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "main.py", line 5, in <module>
    Crawler(target).start()
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/core.py", line 45, in start
    self.__download_album(album)
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/core.py", line 76, in __download_album
    err = '%s 抛出了异常: %s' % (WeiboApi.make_large_url(large), exc)
  File "/Users/Lodour/GitHub/Weibo-Album-Crawler/weibo/api.py", line 132, in make_large_url
    host, name = large_pic['pic_host'], large_pic['pic_name']
TypeError: 'NoneType' object is not subscriptable