search

oGsLP / kuaishou-crawler

As you can see, a kuaishou crawler
MIT License
181 stars 66 forks source link

快手限制下载视频的数量 #28

Open MYF12 opened 3 years ago

MYF12 commented 3 years ago

当爬取数量达到几十个视频之后，就会出现下面的报错，等了很久之后才能继续下载。而且尝试过使用其他ip来继续请求也没用

| kuaishou-crawler (v0.5.0 20-08-06) | 本程序由oGsLP提供, www.github.com/oGsLP/kuaishou-crawler, 喜欢的话可以给个star >_<

准备开始爬取，共有1个用户...

{"data":{"privateFeeds":{"pcursor":"","list":[],"__typename":"PCProfileFeeds"}}}

[] Traceback (most recent call last): File "D:/python_project/test1/crawlers/main.py", line 26, in main() File "D:/python_project/test1/crawlers/main.py", line 20, in main kuaishou.crawler_kuaishou.main() File "D:\python_project\test1\crawlers\kuaishou\crawler_kuaishou.py", line 30, in main crawl(param_did,data_dir) File "D:\python_project\test1\crawlers\kuaishou\crawler_kuaishou.py", line 15, in crawl crawler.crawl() File "D:\python_project\test1\crawlers\kuaishou\lib\crawler.py", line 81, in crawl self.crawl_user(uid) File "D:\python_project\test1\crawlers\kuaishou\lib\crawler.py", line 106, in crawl_user if works[0]['id'] is None: IndexError: list index out of range

MYF12 commented 3 years ago

请问这是什么原因呢？

oGsLP commented 3 years ago

现在很容易被检测出来限制了

MYF12 commented 3 years ago

请问有解决方案吗？比如通过ip池去请求之类的。

oGsLP commented 3 years ago

理论上可行，或者可以先保存json下来，或者保存到数据库，然后一个个爬，被限制了，接着之前的爬就行

MYF12 commented 3 years ago

你有试过通过代理爬取吗？

oGsLP commented 3 years ago

没有