[BUG] - Response 403 - Githubissues

rocailler commented 4 years ago

Since yesterday i have a problem with few endpoints. I've tested only getby endpoints and byHashtag and by Sound have responses 403

>>> api.byHashtag('love')

{'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36', 'accept-encoding': 'gzip, deflate, br', 'accept': 'application/json, text/plain, */*', 'Connection': 'keep-alive', 'authority': 'm.tiktok.com', 'method': 'GET', 'path': '/share/item/list?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F84.0.4147.125+Safari%2F537.36&cookie_enabled=true&screen_width=2560&screen_height=1440&browser_language=&browser_platform=&browser_name=&browser_version=&browser_online=true&timezone_name=&priority_region=&appId=1233&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=822148675&region=US&secUid=&id=4231&type=3&count=30&minCursor=0&maxCursor=0&shareUid=&recType=&lang=en&verifyFp=AaN99VturDQ2&_signature=_02B4Z6wo00f01SDv8tgAAIBB6rMaD8GoX..ZAABdw74', 'scheme': 'https', 'accept-language': 'en-US,en;q=0.9', 'referrer': 'https://www.tiktok.com/', 'sec-fetch-dest': 'empty', 'sec-fetch-mode': 'cors', 'sec-fetch-site': 'same-site'}
Converting response to JSON failed response is below (probably empty)
Traceback (most recent call last):
  File "~/PycharmProjects/backup_scrapers/vevn/lib/python3.8/site-packages/TikTokApi/tiktok.py", line 85, in getData
    return r.json()
  File "~/PycharmProjects/backup_scrapers/vevn/lib/python3.8/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "~/PycharmProjects/backup_scrapers/vevn/lib/python3.8/site-packages/TikTokApi/tiktok.py", line 357, in byHashtag
    res = self.getData(api_url, b, proxy=proxy, language=language)
  File "~/PycharmProjects/backup_scrapers/vevn/lib/python3.8/site-packages/TikTokApi/tiktok.py", line 91, in getData
    raise Exception('Invalid Response')
Exception: Invalid Response

is this a bigger problem or something went wrong on my project?

Desktop:

OS: macOS Catalina
TikTokApi Version 3.4.3/3.4.6

issue-label-bot[bot] commented 4 years ago

Issue-Label Bot is automatically applying the label bug to this issue, with a confidence of 0.96. Please mark this comment with :thumbsup: or :thumbsdown: to give our bot feedback!

Links: app homepage, dashboard and code for this bot.

davidteather commented 4 years ago

It seems to be a larger issue

Tests Failing

byHashtag
bySound
userPager (nightly method)

fastandslow007 commented 4 years ago

Endpoint discoverHashtags still works, but yes byHashtag got exception error.

{'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.125 Safari/537.36', 'accept-encoding': 'gzip, deflate, br', 'accept': 'application/json, text/plain, */*', 'Connection': 'keep-alive', 'authority': 'm.tiktok.com', 'method': 'GET', 'path': '/share/item/list?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F84.0.4147.125+Safari%2F537.36&cookie_enabled=true&screen_width=800&screen_height=600&browser_language=&browser_platform=&browser_name=&browser_version=&browser_online=true&timezone_name=&priority_region=&appId=1233&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=145410044&region=US&secUid=&id=62520&type=3&count=300&minCursor=0&maxCursor=0&shareUid=&recType=&lang=id&verifyFp=DFAQEbKvEsBdxSvt&_signature=_02B4Z6wo00f01orqPowAAIBAs-JmTEY6BGqK-joAAP3-00', 'scheme': 'https', 'accept-language': 'en-US,en;q=0.9', 'referrer': 'https:/www.tiktok.com/', 'sec-fetch-dest': 'empty', 'sec-fetch-mode': 'cors', 'sec-fetch-site': 'same-site'}
Converting response to JSON failed response is below (probably empty)

2020-09-07 08:49:50,218 tiktok [31741] [ERROR]   ERROR-2 Invalid Response
Traceback (most recent call last):
  File "/home/tiktok/.venv_py36/lib/python3.6/site-packages/TikTokApi/tiktok.py", line 85, in getData
    return r.json()
  File "/home/tiktok/.venv_py36/lib/python3.6/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/tiktok/tiktok.py", line 245, in <module>
    tk_agent.start()
  File "/home/tiktok/tiktok.py", line 232, in start
    self.by_hashtag(hashtag=kwd, count=self.post_per_request)
  File "/home/tiktok/tiktok.py", line 125, in by_hashtag
    posts = self.api.byHashtag(hashtag=hashtag, count=count, language='id')
  File "/home/tiktok/.venv_py36/lib/python3.6/site-packages/TikTokApi/tiktok.py", line 357, in byHashtag
    res = self.getData(api_url, b, proxy=proxy, language=language)
  File "/home/tiktok/.venv_py36/lib/python3.6/site-packages/TikTokApi/tiktok.py", line 91, in getData
    raise Exception('Invalid Response')
Exception: Invalid Response

benthinker commented 4 years ago

@davidteather can you share with us possible reasons for this error ? so we can work together to fix it? I am using your TikTok-Api , and using ur interface where i choose 2(for 5-7 min video) and then 2(for hashtag). Thanks!

Granitosaurus commented 4 years ago

There seems to be some API changes for these endpoints.

For example for hashtag endpoint, there are two new(?) parameters:

+ priority_region (e.g. TH for Thailand)
+ tt-web-region (e.g. TH for Thailand)
- user_language parameter seems to be gone now.
\~ ~~max_cursor~~ maxCursor kinda seems to be required to have a proper value now (?), i.e. even for first page

Compared to what I see in chrome devtools.

liamengland1 commented 4 years ago

@Granitosaurus here is the URL of the calls from a guest window. Everything you said seems right. Also, it's maxCursor, not max_cursor.

https://m.tiktok.com/share/item/list?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F85.0.4183.83+Safari%2F537.36&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=en-US&browser_platform=Win32&browser_name=Mozilla&browser_version=5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F85.0.4183.83+Safari%2F537.36&browser_online=true&ac=4g&timezone_name=America%2FNew_York&priority_region=&appId=1233&region=US&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=6869796811135485445&secUid=&id=229207&type=3&count=30&minCursor=0&maxCursor=0&shareUid=&recType=&lang=&verifyFp=verify_kessja7g_BCk1nsDe_YsYQ_4hyP_9bGX_fj1ypHrqLwx0&_signature=_02B4Z6wo00f01YYw7LgAAIBBwNHmb-qJuVWGMegAAD7ec2

Scrolling down: https://m.tiktok.com/share/item/list?aid=1988&app_name=tiktok_web&device_platform=web&referer=&user_agent=Mozilla%2F5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F85.0.4183.83+Safari%2F537.36&cookie_enabled=true&screen_width=1920&screen_height=1080&browser_language=en-US&browser_platform=Win32&browser_name=Mozilla&browser_version=5.0+(Windows+NT+10.0%3B+Win64%3B+x64)+AppleWebKit%2F537.36+(KHTML,+like+Gecko)+Chrome%2F85.0.4183.83+Safari%2F537.36&browser_online=true&ac=4g&timezone_name=America%2FNew_York&priority_region=&appId=1233&region=US&appType=m&isAndroid=false&isMobile=false&isIOS=false&OS=windows&did=6869796811135485445&secUid=&id=229207&type=3&count=30&minCursor=0&maxCursor=30&shareUid=&recType=&lang=&verifyFp=verify_kessja7g_BCk1nsDe_YsYQ_4hyP_9bGX_fj1ypHrqLwx0&_signature=_02B4Z6wo00d015LyXQAAAIBD1BNX1sp3Sg-S81mAALv215

davidteather commented 4 years ago

Thanks for the information, I didn’t have much time to investigate the issue last night. I’ll try to get some more work done on it tonight

davidteather commented 4 years ago

The one thing that is concerning to me is that if you open the request that your browser makes to TikTok in another tab you're given an authorization failed error. I imagine it has to do with the cookies. I'm going to look into it further, but one of my guesses it is the tt_webid_v2 cookie needs to be present as the value of that is also the did parameter in the URL requests

WinkyDev commented 4 years ago

David, I've suggested a fix to the issue. apparently Tiktok decided to check the "Referer" header and it was written as "referrer" in the code. for everyone that need a quick fix like me, just change it on line 76 (tiktok.py)

davidteather commented 4 years ago

@WinkyDev Thank you so much! It seems to be working now. I'll publish a package when the tests start to pass

davidteather / TikTok-Api

[BUG] - Response 403 #251