[GoogleDrive] Support `WEB_EMBEDDED_PLAYER` client for cookieless download links

azowrle commented 9 months ago

DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

[X] I understand that I will be blocked if I intentionally remove or skip any mandatory* field

Checklist

[X] I'm reporting that yt-dlp is broken on a supported site
[X] I've verified that I have updated yt-dlp to nightly or master (update instructions)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched known issues and the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue
[X] I've read about sharing account credentials and I'm willing to share it if required

Region

Australia

Provide a description that is worded well enough to be understood

Hi guys! I am trying to extract JSON data from the Google Drive video link, and when I try to use the format direct URL, the video does not stream directly in the browser. I am just getting: "Access to [endpoint].c.drive.google.com was denied"

Visiting the link and streaming it in the browser using their player works. Also, if I grab the link that their player is using to stream the video from the "Network" tab in Google Chrome and open it in another tab, it will work (stream in the browser).

I think the extractor is broken in some way that does not produce the same link as the one I am getting from the player. Maybe I need to pass some extra args to achieve my goal?

Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[ ] If using API, add 'verbose': True to YoutubeDL params instead
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output

[debug] Command-line config: ['https://drive.google.com/file/d/10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h/view?usp=drive_link', '-J', '-vU']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.12.30 from yt-dlp/yt-dlp [f10589e34] (pip)
[debug] Python 3.11.7 (CPython arm64 64bit) - macOS-14.2.1-arm64-arm-64bit (OpenSSL 3.2.0 23 Nov 2023)
[debug] exe versions: ffmpeg 6.1.1 (setts), ffprobe 6.1.1, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.19.0, brotli-1.1.0, certifi-2023.07.22, mutagen-1.47.0, requests-2.31.0, sqlite3-3.44.2, urllib3-2.2.0, websockets-12.0
[debug] Proxy map: {}
[debug] Request Handlers: urllib, requests, websockets
[debug] Loaded 1798 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: stable@2023.12.30 from yt-dlp/yt-dlp
yt-dlp is up to date (stable@2023.12.30 from yt-dlp/yt-dlp)
[GoogleDrive] Extracting URL: https://drive.google.com/file/d/10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h/view?usp=drive_link
[GoogleDrive] 10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h: Downloading video webpage
[GoogleDrive] 10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h: Requesting source file
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, size, br, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h: Downloading 1 format(s): source
{"id": "10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h", "title": "3. More About Hash Tables.mp4", "thumbnail": "https://drive.google.com/thumbnail?id=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h", "duration": 121, "formats": [{"url": "https://rr4---sn-npoe7nlz.c.drive.google.com/videoplayback?expire=1708948277&ei=BFHcZaLhPLuxp84P_qSysAw&ip=xxxx&id=2b64a390b1928247&itag=18&source=webdrive&requiressl=yes&xpc=EghonaK1InoBAQ==&mh=hI&mm=32%2C26&mn=sn-npoe7nlz%2Csn-30a7rner&ms=su%2Conr&mv=m&mvi=4&pl=24&ttl=transient&susc=dr&driveid=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h&app=explorer&eaua=yN3Qcvu7SP4&mime=video/mp4&vprv=1&prv=1&dur=121.022&lmt=1658904227317417&mt=1708937080&fvip=2&subapp=NONE&txp=0011224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cttl%2Csusc%2Cdriveid%2Capp%2Ceaua%2Cmime%2Cvprv%2Cprv%2Cdur%2Clmt&sig=AJfQdSswRQIhALRAzJyHX74K3fwHDCPlqfBEoJhhBFlVvS7PiNruSBLXAiBtkvslbpH2BXX9XmySbZvosT3Nto7QKxgqosLQ8MV0Pg==&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl&lsig=APTiJQcwRQIhANc6TZPFQFZo59-yaSggIRt2u9vWGhmFOVJO7kJLG6w2AiB66efXmfRH8g9YH6cDlNz9hR5VJ0w74-FfRVnRNfl3pA==", "format_id": "18", "ext": "mp4", "width": 640, "height": 360, "protocol": "https", "resolution": "640x360", "dynamic_range": "SDR", "aspect_ratio": 1.78, "cookies": "DRIVE_STREAM=ihdde9PlARw; Domain=.drive.google.com; Path=/; Secure", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate"}, "video_ext": "mp4", "audio_ext": "none", "vbr": null, "abr": null, "tbr": null, "format": "18 - 640x360"}, {"url": "https://rr4---sn-npoe7nlz.c.drive.google.com/videoplayback?expire=1708948277&ei=BFHcZaLhPLuxp84P_qSysAw&ip=xxxx&id=2b64a390b1928247&itag=22&source=webdrive&requiressl=yes&xpc=EghonaK1InoBAQ==&mh=hI&mm=32%2C26&mn=sn-npoe7nlz%2Csn-30a7rner&ms=su%2Conr&mv=m&mvi=4&pl=24&ttl=transient&susc=dr&driveid=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h&app=explorer&eaua=yN3Qcvu7SP4&mime=video/mp4&vprv=1&prv=1&dur=121.022&lmt=1658904362059945&mt=1708937080&fvip=2&subapp=NONE&txp=0011224&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cttl%2Csusc%2Cdriveid%2Capp%2Ceaua%2Cmime%2Cvprv%2Cprv%2Cdur%2Clmt&sig=AJfQdSswRAIgQAohjoYLb6yoP17oGt57THl2ENMrMLaSy1uiCgpBEugCIF7DfNw2FkYGo-MZ0C71KfTL0yGOVDXmK2yMwnXckLoV&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl&lsig=APTiJQcwRgIhAOvwN3CpLnbttVJhj5Y3p-DnU0wnCv08U6WhKxeWhPd6AiEAkSB_Y3zqxRHqbJd2dWeNqqHLl2rEDGc2nZeeAJhJAmo=", "format_id": "22", "ext": "mp4", "width": 1280, "height": 720, "protocol": "https", "resolution": "1280x720", "dynamic_range": "SDR", "aspect_ratio": 1.78, "cookies": "DRIVE_STREAM=ihdde9PlARw; Domain=.drive.google.com; Path=/; Secure", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate"}, "video_ext": "mp4", "audio_ext": "none", "vbr": null, "abr": null, "tbr": null, "format": "22 - 1280x720"}, {"url": "https://drive.usercontent.google.com/download?id=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h&export=download", "ext": "mp4", "format_id": "source", "quality": 1, "protocol": "https", "resolution": null, "dynamic_range": "SDR", "aspect_ratio": null, "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate"}, "video_ext": "mp4", "audio_ext": "none", "vbr": null, "abr": null, "tbr": null, "format": "source - unknown"}], "subtitles": {}, "automatic_captions": {}, "webpage_url": "https://drive.google.com/file/d/10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h/view?usp=drive_link", "original_url": "https://drive.google.com/file/d/10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h/view?usp=drive_link", "webpage_url_basename": "view", "webpage_url_domain": "drive.google.com", "extractor": "GoogleDrive", "extractor_key": "GoogleDrive", "playlist": null, "playlist_index": null, "thumbnails": [{"url": "https://drive.google.com/thumbnail?id=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h", "id": "0"}], "display_id": "10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h", "fulltitle": "3. More About Hash Tables.mp4", "duration_string": "2:01", "release_year": null, "requested_subtitles": null, "_has_drm": null, "epoch": 1708937480, "requested_downloads": [{"url": "https://drive.usercontent.google.com/download?id=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h&export=download", "ext": "mp4", "format_id": "source", "quality": 1, "protocol": "https", "dynamic_range": "SDR", "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate"}, "video_ext": "mp4", "audio_ext": "none", "format": "source - unknown", "_filename": "3. More About Hash Tables.mp4 [10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h].mp4", "filename": "3. More About Hash Tables.mp4 [10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h].mp4", "__write_download_archive": false}], "url": "https://drive.usercontent.google.com/download?id=10ho0jlCpUkecMUPx0WTAx8PsJ6kNaV0h&export=download", "ext": "mp4", "format_id": "source", "quality": 1, "protocol": "https", "resolution": null, "dynamic_range": "SDR", "aspect_ratio": null, "http_headers": {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/94.0.4606.54 Safari/537.36", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-us,en;q=0.5", "Sec-Fetch-Mode": "navigate"}, "video_ext": "mp4", "audio_ext": "none", "vbr": null, "abr": null, "tbr": null, "format": "source - unknown", "_type": "video", "_version": {"version": "2023.12.30", "current_git_head": null, "release_git_head": "f10589e3453009bb523f55849bba144c9b91cf2a", "repository": "yt-dlp/yt-dlp"}}

bashonly commented 9 months ago

The Google Drive website sets a session cookie that grants you access to the direct download link. When you find the link in your browser's dev tools, the cookie has been set in your browser, so the direct download link works.

When you copy the link from yt-dlp, your browser does not have the cookie set that yt-dlp has. From the JSON output in your log:

"cookies": "DRIVE_STREAM=ihdde9PlARw; Domain=.drive.google.com; Path=/; Secure"

azowrleo commented 9 months ago

The Google Drive website sets a session cookie that grants you access to the direct download link. When you find the link in your browser's dev tools, the cookie has been set in your browser, so the direct download link works.

When you copy the link from yt-dlp, your browser does not have the cookie set that yt-dlp has. From the JSON output in your log:
"cookies": "DRIVE_STREAM=ihdde9PlARw; Domain=.drive.google.com; Path=/; Secure"

Hey thank you for your answer. I did try before and I can confirm that by me setting the cookies I can get a successful response. My issue was the need to set the cookies in the first place.

Grabbing the URL from my network tab and pasting it in a new incognito window does work without having any cookies. Even looking at the URL is very different from the one that I am getting from yt-dlp as it has some additional parameters:

"&cpn=oyjwgn5ZeEVrcnyn&c=WEB_EMBEDDED_PLAYER&cver=1.20240220.01.00"

So I am guessing yt-dlp does not extract the URL the player is using, thus it makes it unplayable in the browser.

Maybe I can pass some additional parameters to yt-dlp to make the URL playable without the need for cookies?

bashonly commented 9 months ago

Grabbing the URL from my network tab and pasting it in a new incognito window does work without having any cookies. Even looking at the URL is very different from the one that I am getting from yt-dlp as it has some additional parameters: "&cpn=oyjwgn5ZeEVrcnyn&c=WEB_EMBEDDED_PLAYER&cver=1.20240220.01.00"

Ah, my bad, I was unaware. Interesting

Maybe I can pass some additional parameters to yt-dlp to make the URL playable without the need for cookies?

No, you would need to edit/rewrite the extractor (or write an extractor override plugin)

yt-dlp / yt-dlp