TikTok extractor no longer working

bandicootdancing commented 5 years ago

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2019.04.01. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[X] I've verified and I assure that I'm running youtube-dl 2019.04.01

Before submitting an issue make sure you have:

[X] At least skimmed through the README, most notably the FAQ and BUGS sections
[X] Searched the bugtracker for similar issues including closed ones
[X] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

What is the purpose of your issue?

[X] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

youtube-dl -v http://vm.tiktok.com/e3a494
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'http://vm.tiktok.com/e3a494']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.04.01
[debug] Python version 2.7.15rc1 (CPython) - Linux-4.15.0-45-generic-x86_64-with-Ubuntu-18.04-bionic
[debug] exe versions: ffmpeg 3.4.4, ffprobe 3.4.4, phantomjs 2.1.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] e3a494: Requesting header
[redirect] Following redirect to https://www.tiktok.com/share/user/79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789
[generic] 79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789: Downloading webpage
[generic] 79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789: Extracting information
ERROR: Unsupported URL: https://www.tiktok.com/share/user/79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2337, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/bin/youtube-dl/youtube_dl/compat.py", line 2551, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/bin/youtube-dl/youtube_dl/compat.py", line 2540, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1659, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1523, in _raiseerror
    raise err
ParseError: syntax error: line 1, column 0
Traceback (most recent call last):
  File "/usr/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 794, in extract_info
    ie_result = ie.extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/common.py", line 529, in extract
    ie_result = self._real_extract(url)
  File "/usr/bin/youtube-dl/youtube_dl/extractor/generic.py", line 3320, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://www.tiktok.com/share/user/79093015200?u_code=d4f5id58ih06h6&language=en&timestamp=1554477103&user_id=6654663938946678789
...
<end of log>

Description of your issue, suggested solution and other information

It looks like tiktok changed their redirect URL for profiles and videos. The extractor no longer works. I did verify the profiles and videos are still there but getting "Error: Unsupported URL" for the new tiktok urls.

jmraker commented 5 years ago

I noticed that there's a few file naming weirdness with tiktok's newer user video profile pages. Not sure if it deserves a new issue as it's all filename related.

It correctly downloads a video but with the extension ".unknown_video" where when it's renamed to a .mp4 file it plays.

When it downloads this url "https://www.tiktok.com/share/video/6675989091407891717?refer=pc_reflow_user" it adds "refer.pc_reflow_user" to the filename. The new url parameters are from clicking on a video at https://www.tiktok.com/share/user/6567659045795758085 in a web browser.

Another different and probably unnecessary file naming behavior from before is it adding " on TikTok" in the filename. It's adding that to every user video page I've tried today. For example 'Chris Alexander on TikTok-6673921230787071238refer=pc_reflow_user.unknown_video'

$ youtube-dl.exe -v "https://www.tiktok.com/share/video/6675989091407891717?refer=pc_reflow_user" [debug] System config: [] [debug] User config: [] [debug] Custom config: [] [debug] Command-line args: ['-v', 'https://www.tiktok.com/share/video/6675989091407891717?refer=pc_reflow_user'] [debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252 [debug] youtube-dl version 2019.04.01 [debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7601-SP1 [debug] exe versions: avconv v13_dev0-1440-g34c1133, avprobe v13_dev0-1440-g34c1133, ffmpeg 4.0.2, ffprobe 4.0.2 [debug] Proxy map: {} [generic] 6675989091407891717?refer=pc_reflow_user: Requesting header WARNING: Falling back on generic information extractor. [generic] 6675989091407891717?refer=pc_reflow_user: Downloading webpage [generic] 6675989091407891717?refer=pc_reflow_user: Extracting information [download] Downloading playlist: Tom on TikTok [generic] playlist Tom on TikTok: Collected 1 video ids (downloading 1 of them) [download] Downloading video 1 of 1 [debug] Default format spec: bestvideo+bestaudio/best [debug] Invoking downloader on 'http://v19.muscdn.com/ba54aff5597e5bbd69d1215f1d079884/5ca874d9/video/tos/maliva/tos-maliva-v-0068/4a248f13868e4fff8d5b905f42ff18cc/?rc=M2g8M3B1bXN4bDMzODczM0ApQHRwbndsQG9FNjQ7NTgzNDMzNjc1MzM0NW9AaDN1KUBmM3UpZnNlemhoZGY2NUBsYS9pbG5eLWZfLS0yMTZzczVvI1xsaHFvI0I0MC0vMTIuLjYwLjIxNi06I28jOmEtcSM6YGheK2IranQ6IzAuXg==' [download] Destination: Tom on TikTok-6675989091407891717refer=pc_reflow_user.unknown_video [download] 100% of 3.32MiB in 00:02 [download] Finished downloading playlist: Tom on TikTok

bandicootdancing commented 5 years ago

Yeah, looks like individual videos are somewhat working like you said. Using "%(uploader)s.%(upload_date)s.%(resolution)s.%(id)s.%(ext)s" still produces those odd filenames. Doesn't seem to be picking up uploader, upload date, or resolutions either.

I have a list of profiles I had set to download once a day so this is a bummer. I can manually grab things for a bit but hopefully this can be figured out soon.

ytdl-org / youtube-dl