Lodour / Weibo-Album-Crawler

A multiprocessing crawler for weibo albums.
MIT License
99 stars 34 forks source link

JSONDecodeError: Expecting value #10

Open unanan opened 6 years ago

unanan commented 6 years ago

The detailed Log:

runfile('E:/Projects/_PYTHON/Weibo-Album-Crawler/main.py', wdir='E:/Projects/_PYTHON/Weibo-Album-Crawler')
[02:05:21] https://weibo.com/*****?refer_flag=100103****_
[02:05:23] 开始下载 "****" 的微博相册
Traceback (most recent call last):

  File "<ipython-input-1-0aaa75bfd107>", line 1, in <module>
    runfile('E:/Projects/_PYTHON/Weibo-Album-Crawler/main.py', wdir='E:/Projects/_PYTHON/Weibo-Album-Crawler')

  File "D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)

  File "D:\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "E:/Projects/_PYTHON/Weibo-Album-Crawler/main.py", line 9, in <module>
    Crawler(target).start()

  File "E:\Projects\_PYTHON\Weibo-Album-Crawler\weibo\core.py", line 39, in start
    total, album_list = WeiboApi.fetch_album_list(self.uid, page, page_size)

  File "E:\Projects\_PYTHON\Weibo-Album-Crawler\weibo\api.py", line 95, in fetch_album_list
    data = WeiboApi.get_json(Url.ALBUM_LIST, params=params)

  File "E:\Projects\_PYTHON\Weibo-Album-Crawler\weibo\api.py", line 67, in get_json
    new_string=json.loads(string)

  File "D:\Anaconda3\lib\json\__init__.py", line 319, in loads
    return _default_decoder.decode(s)

  File "D:\Anaconda3\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())

  File "D:\Anaconda3\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value

I'm new in crawling images on Weibo. So I guess it is because the string I sent to the method json.loads is wrong. I can't find a solution that really works. Can you help me on solving this? Thx in advance.

Lodour commented 6 years ago

@UnaPlugin I'm not so sure what had happened, maybe you could print out the string passed to json.loads and see why it's unexpected.

CharlesLiu7 commented 6 years ago

@unaplugin I have the same issue, the string passed to the json.loads is File not found. And I found on 08/06/2018 that weibo.com doesn't support login only with cookies, but with headers which contains cookies. It works fine. I create a pr or you can also go to my forked repo to see how it works.

Lodour commented 6 years ago

@CharlesLiu7 Sorry, but I couldn't reproduce the login issue as you described. As for #11, maybe the Host field in your header was occasionally verified due to unknown reasons, since requests has provided a default header and will automatically set the Cookie field with the value passed to cookies.

CharlesLiu7 commented 5 years ago

@unaplugin The reason that I can reproduce your problem is that I am still not login with my cookies string. And I find out that cookies string with SCF, SSOLoginState, SUB, SUHB values is enough for me. As for #11, now I think it is useless.