Lodour / Weibo-Album-Crawler

A multiprocessing crawler for weibo albums.
MIT License
99 stars 34 forks source link

下载频率过高会导致API被限制 #15

Closed zhengyongtao closed 4 years ago

zhengyongtao commented 5 years ago

现阶段微博似乎调整了API限制,下载请求过于频繁时会导致获取不到oid,程序会报获取 'oid' 失败,可能原因为 Cookies 过期,或 API 发生变化,等待 60s 后重试。,大概是需要调整一下下载频率

Lodour commented 5 years ago

一般来说持续几分钟后会自动恢复,可以挂着不用管

zhengyongtao commented 5 years ago

没有恢复,于是我将配置中的COOKIE配置成AUTO,结果遇到了另一个问题,当chromedriver打开微博页面之后,就没有下一步操作了,这部分文档里没有讲清楚后续应该怎么操作,我尝试用扫码登录没什么用。一段时间后会报超时错误。错误如下

Traceback (most recent call last):
  File "F:/Project/Weibo/main.py", line 1, in <module>
    from weibo.core import Crawler
  File "F:\Project\Weibo\weibo\core.py", line 11, in <module>
    from weibo.api import WeiboApi
  File "F:\Project\Weibo\weibo\api.py", line 39, in <module>
    class WeiboApi(object):
  File "F:\Project\Weibo\weibo\api.py", line 45, in WeiboApi
    COOKIES = _load_cookies()
  File "F:\Project\Weibo\weibo\api.py", line 34, in _load_cookies
    return weibo.get_cookies()
  File "F:\Project\Weibo\weibo\browser.py", line 16, in get_cookies
    qrcode_tab = self._wait_element('//a[@node-type="qrcode_tab"]')
  File "F:\Project\Weibo\weibo\browser.py", line 29, in _wait_element
    element = WebDriverWait(self._driver, 60).until(condition)
  File "F:\Project\Weibo\venv\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: