[Broken]Tiktok seems to be broken

someziggyman commented 3 years ago

Checklist

[x] I'm reporting a broken site support
[x] I've verified that I'm running youtube-dlc version 2020.10.24
[x] I've checked that all provided URLs are alive and playable in a browser
[x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[x] I've searched the bugtracker for similar issues including closed ones

Verbose log

ytdlc -v https://www.tiktok.com/@charlidamelio/video/6854565358808993029
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.tiktok.com/@charlidamelio/video/6854565358808993029']
[debug] Loading archive file None
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dlc version 2020.10.24-5
[debug] Python version 2.7.16 (CPython) - Darwin-19.6.0-x86_64-i386-64bit
[debug] exe versions: none
[debug] Proxy map: {}
[TikTok] 6854565358808993029: Downloading video webpage
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "./ytdlc/__main__.py", line 19, in <module>
  File "./ytdlc/youtube_dlc/__init__.py", line 487, in main
  File "./ytdlc/youtube_dlc/__init__.py", line 477, in _real_main
  File "./ytdlc/youtube_dlc/YoutubeDL.py", line 2096, in download
  File "./ytdlc/youtube_dlc/YoutubeDL.py", line 830, in extract_info
  File "./ytdlc/youtube_dlc/extractor/common.py", line 532, in extract
  File "./ytdlc/youtube_dlc/extractor/tiktok.py", line 137, in _real_extract
  File "./ytdlc/youtube_dlc/extractor/tiktok.py", line 22, in _extract_aweme
AttributeError: 'NoneType' object has no attribute 'get'

Description

https://www.tiktok.com/@charlidamelio/video/6854565358808993029

merval commented 3 years ago

The JSON response changed, which lead to various things no longer pointing where they should have. If the PR is merged, it'll fix it.

someziggyman commented 3 years ago

Cool. Thanks for your help. It seems to extract video format correctly, but if I try to download it, I'm getting 403: Forbidden error

./ytdlc -v https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: [u'-v', u'https://www.tiktok.com/@charlidamelio/video/6854565358808993029'] [debug] Loading archive file None [debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8 [debug] youtube-dlc version 2020.10.24-5 [debug] Python version 2.7.16 (CPython) - Darwin-19.6.0-x86_64-i386-64bit [debug] exe versions: none [debug] Proxy map: {} [TikTok] 6854565358808993029: Downloading video webpage [debug] Default format spec: best/bestvideo+bestaudio [debug] Invoking downloader on u'https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603663286&l=20201025160111010190219210330492C5&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw%3D%3D&signature=bf1f449249ac4ac7f3917437f088d07f&tk=tt_webid_v2&vl=&vr=' ERROR: unable to download video data: HTTP Error 403: Forbidden Traceback (most recent call last): File "./ytdlc/youtube_dlc/YoutubeDL.py", line 2003, in process_info success = dl(filename, info_dict) File "./ytdlc/youtube_dlc/YoutubeDL.py", line 1861, in dl return fd.download(name, info) File "./ytdlc/youtube_dlc/downloader/common.py", line 366, in download return self.real_download(filename, info_dict) File "./ytdlc/youtube_dlc/downloader/http.py", line 349, in real_download establish_connection() File "./ytdlc/youtube_dlc/downloader/http.py", line 114, in establish_connection raise err HTTPError: HTTP Error 403: Forbidden

johndpknt commented 3 years ago

Did you manage to find any resolution, came accross the same problem

merval commented 3 years ago

Did you manage to find any resolution, came accross the same problem

There is a PR open for this, but there appears to be a permissions issue reading the video URL. Trying to work out a solution

october262 commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029 works with youtube-dl

someziggyman commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029 works with youtube-dl

it works once. then it's the same 403 error.

october262 commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029 works with youtube-dl

it works once. then it's the same 403 error.

try again it worked for me.

merval commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029

works with youtube-dl

it works once. then it's the same 403 error.

try again it worked for me.

I wonder if the TikTok servers are blocking the direct requests...

october262 commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029

works with youtube-dl

it works once. then it's the same 403 error.

try again it worked for me.

I wonder if the TikTok servers are blocking the direct requests...

just tried two more times - first time error, second time = success

someziggyman commented 3 years ago

this link - https://www.tiktok.com/@charlidamelio/video/6854565358808993029 works with youtube-dl

it works once. then it's the same 403 error.

try again it worked for me.

Ok, so here's my log. As you see, with the latest youtube-dl it works, then doesn't, then works, works, doesn't... witchcraft...

[generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676483&l=20201025194108010190176042520889A2&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=25c720f5ba4a352cfae23a8fc2e301c6&tk=tt_webid_v2&vl=&vr=: Requesting header WARNING: Could not send HEAD request to https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676483&l=20201025194108010190176042520889A2&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=25c720f5ba4a352cfae23a8fc2e301c6&tk=tt_webid_v2&vl=&vr=: HTTP Error 403: Forbidden [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676483&l=20201025194108010190176042520889A2&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=25c720f5ba4a352cfae23a8fc2e301c6&tk=tt_webid_v2&vl=&vr=: Downloading webpage ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

youtube-dl -F https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676528&l=2020102519415301018919607046082004&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=67dcb74f11d04827985d61a4d5452816&tk=tt_webid_v2&vl=&vr=: Requesting header WARNING: Could not send HEAD request to https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676528&l=2020102519415301018919607046082004&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=67dcb74f11d04827985d61a4d5452816&tk=tt_webid_v2&vl=&vr=: HTTP Error 403: Forbidden [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676528&l=2020102519415301018919607046082004&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=67dcb74f11d04827985d61a4d5452816&tk=tt_webid_v2&vl=&vr=: Downloading webpage ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

youtube-dl https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [download] Downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness [generic] playlist momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness: Collected 1 video ids (downloading 1 of them) [download] Downloading video 1 of 1 [download] Destination: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness-6854565358808993029.unknown_video [download] 100% of 3.46MiB in 00:00 [download] Finished downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness

youtube-dl https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676695&l=2020102519444001018905603421088380&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=7f7acc83c417e285c48a9ca28e6319bf&tk=tt_webid_v2&vl=&vr=: Requesting header WARNING: Could not send HEAD request to https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676695&l=2020102519444001018905603421088380&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=7f7acc83c417e285c48a9ca28e6319bf&tk=tt_webid_v2&vl=&vr=: HTTP Error 403: Forbidden [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676695&l=2020102519444001018905603421088380&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=7f7acc83c417e285c48a9ca28e6319bf&tk=tt_webid_v2&vl=&vr=: Downloading webpage ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

$ youtube-dl https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [download] Downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness [generic] playlist momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness: Collected 1 video ids (downloading 1 of them) [download] Downloading video 1 of 1 [download] momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness-6854565358808993029.unknown_video has already been downloaded [download] 100% of 3.46MiB [download] Finished downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness

youtube-dl https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [download] Downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness [generic] playlist momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness: Collected 1 video ids (downloading 1 of them) [download] Downloading video 1 of 1 [download] momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness-6854565358808993029.unknown_video has already been downloaded [download] 100% of 3.46MiB [download] Finished downloading playlist: momma @heididamelio dc @trvpandre @dejuane.mccoy @justinplaness

youtube-dl https://www.tiktok.com/@charlidamelio/video/6854565358808993029 [generic] 6854565358808993029: Requesting header WARNING: Falling back on generic information extractor. [generic] 6854565358808993029: Downloading webpage [generic] 6854565358808993029: Extracting information [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676713&l=202010251944580101880612193A086626&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=5e96428ca0a5f84ec9a141ca0537d09d&tk=tt_webid_v2&vl=&vr=: Requesting header WARNING: Could not send HEAD request to https://v16-web.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/c1c3a0d1ef7848899a4bca12777bafc4/?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676713&l=202010251944580101880612193A086626&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=5e96428ca0a5f84ec9a141ca0537d09d&tk=tt_webid_v2&vl=&vr=: HTTP Error 403: Forbidden [generic] ?a=1988&br=3754&bt=1877&cr=0&cs=0&cv=1&dr=0&ds=3&er=&expire=1603676713&l=202010251944580101880612193A086626&lr=tiktok_m&mime_type=video_mp4&policy=2&qs=0&rc=amo6Z2w8c3M3djMzZzczM0ApZzdkODdnOGRlNzxpZWU5aWc2b2lec2QxcWRfLS01MTZzcy8zLTRfXzQxL15hNmEvLjQ6Yw==&signature=5e96428ca0a5f84ec9a141ca0537d09d&tk=tt_webid_v2&vl=&vr=: Downloading webpage ERROR: Unable to download webpage: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type youtube-dl -U to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

someziggyman commented 3 years ago

Hey guys, for some reason I'm still getting that 403: forbidden error. Could this be because I'm running this on MacOS terminal with Python version 2.7.16 ? using the latest release of dlc.

merval commented 3 years ago

I don't think it has anything to do with your python version. I've got the same issue here. Even though the cookies are in the header, it fails. running with --cookies cookie.txt, the only cookies are the two added to the headers. So weird. @blackjack4494

sazo009 commented 3 years ago

I am also getting 403 errors. Just making some observations here:

If I look at a video page loaded in a web browser, it's setting 4 cookies. If I make a python request to this same page, I only get 2 cookies set: tt_webid and tt_webid_v2. The missing ones are MONITOR_WEB_ID and s_v_web_id.

Share links do actually consistently work. As in vm.tiktok.com/something. You generate those from inside the app. yt-dlc will follow a redirect from the share link to a normal-looking video URL with a number of extra parameters passed in. Not sure what else is different.

merval commented 3 years ago

I am also getting 403 errors. Just making some observations here:

If I look at a video page loaded in a web browser, it's setting 4 cookies. If I make a python request to this same page, I only get 2 cookies set: tt_webid and tt_webid_v2. The missing ones are MONITOR_WEB_ID and s_v_web_id.

Share links do actually consistently work. As in vm.tiktok.com/something. You generate those from inside the app. yt-dlc will follow a redirect from the share link to a normal-looking video URL with a number of extra parameters passed in. Not sure what else is different.

If you add --cookies cookies.txt and then run it a second time, it'll work. I don't think it's a missing cookie issue, since the cookies written are just those two.

GreyAlien502 commented 3 years ago

Also, i've submitted pull request #20 for the 403 error. Any feedback would be welcome. It always works for me, but the more testing the better.

blackjack4494 / yt-dlc