JustAnotherArchivist / snscrape

A social networking service scraper in Python
GNU General Public License v3.0
4.43k stars 706 forks source link

when get_items found all data. #964

Closed SpringBoom closed 1 year ago

SpringBoom commented 1 year ago

When I use WeiboUserScraper.get_items():

Error retrieving https://xxxx...: no-content message
4 requests to https://xxxx... failed, giving up.
Errors: no-content message, no-content message, no-content message, no-content message

But all data is obtained. Retries when response is no data:

def _check_timeline_response(self, r):
if r.status_code == 200 and r.content == b'{"ok":0,"msg":"\\u8fd9\\u91cc\\u8fd8\\u6ca1\\u6709\\u5185\\u5bb9","data":{"cards":[]}}':
    # 'No content here yet'. Appears to happen sometimes on pagination, possibly due to too fast requests; retry this
    return False, 'no-content message'
if r.status_code != 200:
    return False, 'non-200 status code'
return True, None

So, How to set break get_items when it's really no data in response?

JustAnotherArchivist commented 1 year ago

This would be a bug, not a question. Please use the correct issue template and include a complete reproducing example (i.e. a user, ideally one with only a few posts).