search

yt-dlp / yt-dlp

A feature-rich command-line audio/video downloader
https://discord.gg/H5MNcFW63r
The Unlicense
73.35k stars 5.81k forks source link

[twitter] x.com links give Unsupported URL error #9923

Closed InklingGirl closed 2 weeks ago

InklingGirl commented 2 weeks ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

Checklist

Region

Australia

Provide a description that is worded well enough to be understood

https://twitter.com has been changed to https://x.com, but yt-dlp is still performing a redirect to the former URL it does not need to, which seems to break the download.

Provide verbose output that clearly demonstrates the problem

Complete Verbose Output

[debug] Command-line config: ['-vU', 'https://x.com/historyinmemes/status/1790637656616943991']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2024.04.09 from yt-dlp/yt-dlp [ff0779267] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1810 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2024.04.09 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2024.04.09 from yt-dlp/yt-dlp)
[generic] Extracting URL: https://x.com/historyinmemes/status/1790637656616943991
[generic] 1790637656616943991: Downloading webpage
[redirect] Following redirect to https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
[generic] Extracting URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
[generic] migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P: Downloading webpage
WARNING: [generic] Falling back on generic information extractor
[generic] migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P: Extracting information
[debug] Looking for embeds
ERROR: Unsupported URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
Traceback (most recent call last):
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1606, in wrapper
    return func(self, *args, **kwargs)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/YoutubeDL.py", line 1741, in __extract_info
    ie_result = ie.extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/generic.py", line 2514, in _real_extract
    raise UnsupportedError(url)
yt_dlp.utils.UnsupportedError: Unsupported URL: https://twitter.com/x/migrate?tok=eyJlIjoiL2hpc3Rvcnlpbm1lbWVzL3N0YXR1cy8xNzkwNjM3NjU2NjE2OTQzOTkxIiwidCI6MTcxNTc2OTk4MH26emBettwJjp9vBvDz7z1P
joe-oli commented 2 weeks ago

I confirm yt-dlp no longer works on X/Twitter; just started happening very recently (approximately 14/May/2024), which is most likely related to the url for X is now x.com instead of twitter.com

bashonly commented 2 weeks ago

Internally, the site still uses twitter domains. All we need to do is add x.com to the _VALID_URLs

InklingGirl commented 2 weeks ago

Will this update be pushed to pip soon?

bashonly commented 2 weeks ago

@InklingGirl already has: python -m pip install -U --pre "yt-dlp[default]"

jshir commented 2 weeks ago

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _____: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

ggous commented 1 week ago

I have updated : sudo yt-dlp -U

and I am receiving:

ERROR: Unsupported URL: https://x.com/i/status/1792192732015800675

bashonly commented 1 week ago

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

ggous commented 1 week ago

@ggous

you need to have updated to nightly

yt-dlp --update-to nightly

If it still doesn't work, then show a verbose log: add -v to your download command and copy/paste the full output

Thanks! It works now!

hsn10 commented 1 week ago

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

Grub4K commented 1 week ago

its high priority bugfix because x.com is very popular site, probably second only to youtube

new yt-dlp version should be published.

Use yt-dlp --update-to nightly, download the nightly build manually or use pip install --pre "yt-dlp[default]".

InklingGirl commented 1 week ago

I installed the nightly and public videos work fine, however, I cannot get it to download a protected video by passing a cookies.txt file from x.com (this is after a very recent change to their login system). I bet I'm not the only one affected and I don't want to give out someone's private link so here is just the error message without the verbose. Let me know if a verbose is truly necessary and I would have to set up a dummy protected acct to get one.

ERROR: [twitter] _____: You are not authorized to view this protected tweet. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials

same here @jshir

bashonly commented 1 week ago

show verbose log please @InklingGirl

InklingGirl commented 1 week ago

show verbose log please @InklingGirl

@bashonly this happens w/ any video that previously required & worked w/ cookies:


[debug] Command-line config: ['--cookies', 'Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.16.232713 from yt-dlp/yt-dlp-nightly-builds [6d8a53d87] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1803 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
[debug] Downloading _update_spec from https://github.com/yt-dlp/yt-dlp-nightly-builds/releases/latest/download/_update_spec
Current version: nightly@2024.05.16.232713 from yt-dlp/yt-dlp-nightly-builds
Latest version: nightly@2024.05.23.232707 from yt-dlp/yt-dlp-nightly-builds
ERROR: You installed yt-dlp with pip or using the wheel from PyPi; Use that to update
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1428, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1406, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1283, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1233, in raise_login_required
    raise ExtractorError(msg, expected=True)
bashonly commented 1 week ago

@InklingGirl update. Your (old) nightly version does not have 3e35aa32c74bc108375be8c8b6b3bfc90dfff1b4, which fixed that issue

python3 -m pip install --pre -U "yt-dlp[default]"
InklingGirl commented 1 week ago

I upgraded to it & get the exact same result.

EDIT: I generated a new cookies file & that solved the issue. The old cookies must have been for twitter.com instead of x.com & that was what caused the problem.

InklingGirl commented 4 days ago

& it seems to be broken again.

bashonly commented 4 days ago

@InklingGirl please share a verbose log

InklingGirl commented 4 days ago 

[debug] Command-line config: ['--cookies', '/home/christian/Documents/cookies.txt', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)
bashonly commented 4 days ago

What happens when you try with --cookies-from-browser?

InklingGirl commented 4 days ago

Is that supposed to work w/ an apt-get installation of brave-browser on Ubuntu 22?

bashonly commented 4 days ago

it should yeah

InklingGirl commented 4 days ago 

yt-dlp: error: unsupported browser specified for cookies: "https". Supported browsers are: brave, chrome, chromium, edge, firefox, opera, safari, vivaldi, whale
InklingGirl commented 4 days ago

I'm assuming I had to specify the browser, here's the verbose log for that:


[debug] Command-line config: ['--cookies-from-browser=brave', '-vU', '<MANUALLY_REDACTED_URL>']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds [111b61dde] (pip)
[debug] Python 3.10.12 (CPython x86_64 64bit) - Linux-6.5.0-35-generic-x86_64-with-glibc2.35 (OpenSSL 3.0.2 15 Mar 2022, glibc 2.35)
[debug] exe versions: ffmpeg 4.4.2 (setts), ffprobe 4.4.2, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.11.0, brotli-1.0.9, certifi-2020.06.20, mutagen-1.47.0, pyxattr-0.7.2, requests-2.31.0, secretstorage-3.3.1, sqlite3-3.37.2, urllib3-2.2.1, websockets-12.0
[debug] Proxy map: {}
Extracting cookies from brave
[debug] Extracting cookies from: "/home/christian/.config/BraveSoftware/Brave-Browser/Default/Cookies"
[Cookies] Loading cookie      0/  2019XDG_CURRENT_DESKTOP is set to an unknown value: "ubuntu"
[debug] detected desktop environment: OTHER
[debug] Chosen keyring: BASICTEXT
WARNING: cannot decrypt v11 cookies: no key found
Extracted 44 cookies from brave (1975 could not be decrypted)
[debug] cookie version breakdown: {'v10': 0, 'v11': 1975, 'other': 0, 'unencrypted': 44}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1820 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp-nightly-builds/releases/latest
Latest version: nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds
yt-dlp is up to date (nightly@2024.05.27.232744 from yt-dlp/yt-dlp-nightly-builds)
[twitter] Extracting URL: <MANUALLY_REDACTED_URL>
[twitter] <MANUALLY_REDACTED_URL>: Downloading guest token
[twitter] <MANUALLY_REDACTED_URL>: Downloading GraphQL JSON
ERROR: [twitter] <MANUALLY_REDACTED_URL>: NSFW tweet requires authentication. Use --cookies, --cookies-from-browser, --username and --password, --netrc-cmd, or --netrc (twitter) to provide account credentials
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 734, in extract
    ie_result = self._real_extract(url)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1434, in _real_extract
    status = self._extract_status(twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1412, in _extract_status
    status = self._graphql_to_legacy(self._call_graphql_api(self._GRAPHQL_ENDPOINT, twid), twid)
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/twitter.py", line 1289, in _graphql_to_legacy
    self.raise_login_required('NSFW tweet requires authentication')
  File "/home/christian/.local/lib/python3.10/site-packages/yt_dlp/extractor/common.py", line 1239, in raise_login_required
    raise ExtractorError(msg, expected=True)
bashonly commented 2 days ago

[debug] Extracting cookies from: "/home/christian/.config/BraveSoftware/Brave-Browser/Default/Cookies" [Cookies] Loading cookie 0/ 2019XDG_CURRENT_DESKTOP is set to an unknown value: "ubuntu"

@InklingGirl Something is going wrong with --cookies-from-browser here. It might be worth opening a separate issue about this failure to extract cookies. So for this log, it's not working because yt-dlp isn't getting the cookies it needs.

As for your attempt with the cookies txt file, would you be willing to share a redacted copy of the cookiefile? There's no other way for me to see what's going on with it.

InklingGirl commented 2 days ago

I don't know how exactly to go about doing that at the moment, I wouldn't necessarily know what to redact, but what I can tell you is I am using an apt installation of brave-browser & the file was produced by the F/LOSS Chromium extension Get cookies.txt LOCALLY. @bashonly