ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.96k stars 10.01k forks source link

Vimeo - Embedded / Referrer: Unable to extract info section #31536

Closed Casper042 closed 1 year ago

Casper042 commented 1 year ago

Checklist

Verbose log

casper@wendy:~/Docker/YoutubeDL-Material$ docker exec -it YouTubeDL /app/node_modules/youtube-dl/bin/youtube-dl --add-header Referer:"https://videotextonline.com/" -o /app/video/AlgebraV-B-5 https://player.vimeo.com/video/113302044?fun=0 --verbose
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--add-header', 'Referer:https://videotextonline.com/', '-o', '/app/video/AlgebraV-B-5', 'https://player.vimeo.com/video/113302044?fun=0', '--verbose']
[debug] Adding header from command line option Referer:https://videotextonline.com/
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Python version 3.8.10 (CPython) - Linux-4.15.0-197-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg N-63101-gc92edd969a-static, ffprobe N-63101-gc92edd969a-static
[debug] Proxy map: {}
[vimeo] 113302044: Downloading webpage
ERROR: Unable to extract info section; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/app/node_modules/youtube-dl/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/app/node_modules/youtube-dl/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
    ie_result = ie.extract(url)
  File "/app/node_modules/youtube-dl/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
    ie_result = self._real_extract(url)
  File "/app/node_modules/youtube-dl/bin/youtube-dl/youtube_dl/extractor/vimeo.py", line 651, in _real_extract
    config = self._parse_json(self._search_regex(
  File "/app/node_modules/youtube-dl/bin/youtube-dl/youtube_dl/extractor/common.py", line 1012, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract info section; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

casper@wendy:~/Docker/YoutubeDL-Material$
casper@wendy:~/Docker/YoutubeDL-Material$
casper@wendy:~/Docker/YoutubeDL-Material$ docker exec -it YouTubeDL /app/node_modules/youtube-dl/bin/youtube-dl  --version                        2021.12.17
casper@wendy:~/Docker/YoutubeDL-Material$

Description

Attempting to download my son's Algebra lessons hosted on Vimeo so we can watch them easier on the big screen. Same exact Syntax was used in October 2022 successfully, I even still have a shell script from then I used to batch the requests. Now I get the above error. I'm calling YTDL from inside a Material container because it was convenient, had it already. Updated that container to latest and as per last few lines of Verbose log above, it is indeed internally using YTDL 2021.12.17

Have verified the Vimeo plays fine when embedded on the Course website. Player looks to have slightly changed since last time.

F12 in Chrome from Course website shows the following in case it's helpful: (I see a bunch of "swarmify" stuff I don't recall seeing last time. I was always able to identify the correct Vimeo link before by looking for "fun=0"

<div src="https://player.vimeo.com/video/113927055?fun=0" width="500" height="375" webkitallowfullscreen="" mozallowfullscreen="" class="swarm-iframe smartvideo_4014-iframe-dimensions" style="max-width: 100%;" id="yui_3_17_2_3_1676588444912_56"><div style="max-width: 100%; float: none;" preload="none" playsinline="playsinline" id="smartvideo_4014_videojs" class="smartvideo-player vjs-swarmify-theme fitvidsignore dimensions-smartvideo_4014_videojs vjs-controls-enabled vjs-has-started vjs-paused vjs-user-inactive" allowfullscreen="" mozallowfullscreen="" webkitallowfullscreen="" frameborder="0"><video frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen="" class="vjs-tech" id="smartvideo_4014_videojs_html5_api" playsinline="playsinline" preload="none" style="max-width: 100%; float: none;" src="blob:https://videotextonline.com/e0ff5f5c-c287-4985-b077-ed5370195604"></video><div></div><div class="vjs-poster" tabindex="-1" data-background-image="https://video-node.swarmcdn.com/38c90990-704f-42a9-95cf-ca7fd18a60d4/cec7b43b7573d5dca138f3cec48f3b382a69a5d1efa44fd88a43654b6bd91568.jpg" data-loaded="true" style="background-image: url(&quot;https://video-node.swarmcdn.com/38c90990-704f-42a9-95cf-ca7fd18a60d4/cec7b43b7573d5dca138f3cec48f3b382a69a5d1efa44fd88a43654b6bd91568.jpg&quot;); display: none;"></div><div class="vjs-loading-spinner"></div><div class="vjs-swarmify-disabled" role="button" aria-live="polite" tabindex="0" aria-label="disabled"><div class="vjs-control-content"><span class="vjs-control-text">Need Text</span></div></div><div class="vjs-swarmify-play-button" role="button" aria-live="polite" tabindex="0" aria-label="play video" id="yui_3_17_2_3_1676588444912_55"><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 294 336" class="swarmify-play-button hexagon" id="yui_3_17_2_3_1676588444912_54"><defs><path id="h" d="M158 0l136.83 79v158L158 316 21.168 237V79z"></path><path id="t" d="M53.499 7.951l51.961 90H1.539z"></path></defs><g stroke-linejoin="round" fill-rule="evenodd" stroke-linecap="round" id="yui_3_17_2_3_1676588444912_53"><g id="hex" transform="translate(-11 10)"><use xlink:href="#h" height="100%" width="100%"></use><path stroke-width="10" d="M158-5.774l141.83 81.888v163.77L158 321.774l-141.83-81.88V76.124L158-5.77z"></path></g><g transform="rotate(-30 289.199 -57.894)" id="tri"><use xlink:href="#t" height="100%" width="100%"></use><path stroke-width="4" d="M53.499 3.951l55.424 96H-1.927l55.429-96z" id="yui_3_17_2_3_1676588444912_52"></path></g></g></svg><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 294 336" class="swarmify-play-button rectangle"><defs><path id="h" d="M158 0l136.83 79v158L158 316 21.168 237V79z"></path><path id="t" d="M53.499 7.951l51.961 90H1.539z"></path></defs><g stroke-linejoin="round" fill-rule="evenodd" stroke-linecap="round"><g id="hex"><rect width="280" height="174" x="50%" y="50%" transform="translate(-140,-87)" id="svg_3" stroke-width="10"></rect></g><g transform="rotate(-30 289.199 -57.894)" id="tri"><use xlink:href="#t" height="100%" width="100%"></use><path stroke-width="4" d="M53.499 3.951l55.424 96H-1.927l55.429-96z"></path></g></g></svg><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" viewBox="0 0 294 336" class="swarmify-play-button circle"><defs><path id="h" d="M158 0l136.83 79v158L158 316 21.168 237V79z"></path><path id="t" d="M53.499 7.951l51.961 90H1.539z"></path></defs><g stroke-linejoin="round" fill-rule="evenodd" stroke-linecap="round"><g id="hex"><circle cx="50%" cy="50%" r="125" id="svg_3" stroke-width="10"></circle></g><g transform="rotate(-30 289.199 -57.894)" id="tri"><use xlink:href="#t" height="100%" width="100%"></use><path stroke-width="4" d="M53.499 3.951l55.424 96H-1.927l55.429-96z"></path></g></g></svg></div><div class="vjs-swarmify-watermark vjs-swarmify-watermark-bottom-right" role="button" aria-live="polite" tabindex="0" aria-label="watermark"><a rel="noopener" target="_blank" href="https://swarmify.com/player/learn-more/?utm_campaign=wtrmrk&amp;utm_medium=player&amp;utm_source=videotextonline.com"><img src="https://assets.swarmcdn.com/cross/images/swarmify_logo_grey.png" loading="lazy" width="2265" height="567" alt="Swarmify Video Hosting"></a></div><div class="vjs-control-bar" style=""><div class="vjs-play-control vjs-control  vjs-paused" role="button" aria-live="polite" tabindex="0"><div class="vjs-control-content"><span class="vjs-control-text">Play</span></div></div><div class="vjs-current-time vjs-time-controls vjs-control" style="display: block;"><div class="vjs-current-time-display" aria-live="off"><span class="vjs-control-text">Current Time</span> 00:12</div></div><div class="vjs-time-divider" style="display: block;"><div><span>/</span></div></div><div class="vjs-duration vjs-time-controls vjs-control" style="display: block;"><div class="vjs-duration-display" aria-live="off"><span class="vjs-control-text">Duration Time</span> 11:02</div></div><div class="vjs-remaining-time vjs-time-controls vjs-control" style="display: none;"><div class="vjs-remaining-time-display" aria-live="off"><span class="vjs-control-text">Remaining Time</span> -10:50</div></div><div class="vjs-live-controls vjs-control"><div class="vjs-live-display" aria-live="off"><span class="vjs-control-text">Stream Type</span>LIVE</div></div><div class="vjs-progress-control vjs-control"><div role="slider" aria-valuenow="1.84" aria-valuemin="0" aria-valuemax="100" tabindex="0" class="vjs-progress-holder vjs-slider" aria-label="video progress bar" aria-valuetext="00:12"><div class="vjs-load-progress" style="width: 17.2055%;"><span class="vjs-control-text" style="left: 0%; width: 100%;"><span>Loaded</span>: 0%</span></div><div class="vjs-play-progress" style="width: 1.84%;"><span class="vjs-control-text"><span>Progress</span>: 0%</span></div><div class="vjs-seek-handle vjs-slider-handle" aria-live="off" style="left: 1.84%;"><span class="vjs-control-text">00:12</span></div></div></div><div class="vjs-fullscreen-control vjs-control " role="button" aria-live="polite" tabindex="0"><div class="vjs-control-content"><span class="vjs-control-text">Fullscreen</span></div></div><div class="vjs-swarmify-signal-icon vjs-menu-button vjs-control swarm-on" style="display: block;"><div class="signal-popup vjs-menu"><a rel="noopener" target="_blank" href="https://swarmify.com/player/learn-more/"><div class="vjs-menu-content"><div class="signal-title">Video Acceleration:</div><div class="signal-on-text">On</div><div class="signal-off-text">Off</div><br></div></a></div></div><div class="vjs-volume-control vjs-control"><div role="slider" aria-valuenow="100" aria-valuemin="0" aria-valuemax="100" tabindex="0" class="vjs-volume-bar vjs-slider" aria-label="volume level" aria-valuetext="100%"><div class="vjs-volume-level" style="width: 100%;"><span class="vjs-control-text"></span></div><div class="vjs-volume-handle vjs-slider-handle" style="left: 100%;"><span class="vjs-control-text">00:00</span></div></div></div><div class="vjs-mute-control vjs-control vjs-vol-3" role="button" aria-live="polite" tabindex="0"><div><span class="vjs-control-text">Mute</span></div></div><div class="vjs-playback-rate vjs-menu-button vjs-control" aria-haspopup="true" role="button"><div class="vjs-control-content"><span class="vjs-control-text">Playback Rate</span></div><div class="vjs-playback-rate-value">1x</div><div class="vjs-menu"><ul class="vjs-menu-content"><li class="vjs-menu-item" role="button" aria-live="polite" tabindex="0" aria-selected="false">2x</li><li class="vjs-menu-item" role="button" aria-live="polite" tabindex="0" aria-selected="false">1.5x</li><li class="vjs-menu-item" role="button" aria-live="polite" tabindex="0" aria-selected="false">1.25x</li><li class="vjs-menu-item vjs-selected" role="button" aria-live="polite" tabindex="0" aria-selected="true">1x</li><li class="vjs-menu-item" role="button" aria-live="polite" tabindex="0" aria-selected="false">0.75x</li><li class="vjs-menu-item" role="button" aria-live="polite" tabindex="0" aria-selected="false">0.5x</li></ul></div></div><div class="vjs-subtitles-button vjs-menu-button vjs-control " role="button" aria-live="polite" tabindex="0" aria-haspopup="true" aria-label="Subtitles Menu" style="display: none;"><div class="vjs-control-content"><span class="vjs-control-text">Subtitles</span><div class="vjs-menu"><ul class="vjs-menu-content"><li class="vjs-menu-item vjs-selected" role="button" aria-live="polite" tabindex="0" aria-selected="true">subtitles off</li></ul></div></div></div><div class="vjs-captions-button vjs-menu-button vjs-control " role="button" aria-live="polite" tabindex="0" aria-haspopup="true" aria-label="Captions Menu" style="display: none;"><div class="vjs-control-content"><span class="vjs-control-text">Captions</span><div class="vjs-menu"><ul class="vjs-menu-content"><li class="vjs-menu-item vjs-selected" role="button" aria-live="polite" tabindex="0" aria-selected="true">captions off</li></ul></div></div></div><div class="vjs-chapters-button vjs-menu-button vjs-control " role="button" aria-live="polite" tabindex="0" aria-haspopup="true" aria-label="Chapters Menu" style="display: none;"><div class="vjs-control-content"><span class="vjs-control-text">Chapters</span><div class="vjs-menu"><ul class="vjs-menu-content"><li class="vjs-menu-title">Chapters</li></ul></div></div></div></div><div class="vjs-error-display"><div></div></div></div></div>
Casper042 commented 1 year ago

PS: I found a previous bug from a few years ago with the exact same issue, but that was closed after an update to YTDL resolved the issue. Not a valid option here since I seem to be running latest,

dirkf commented 1 year ago

This works fine with the git master.

$ python -m youtube_dl -v -F 'https://player.vimeo.com/video/113927055?fun=0' --referer 'https://videotextonline.com/'
[debug] System config: [u'--prefer-ffmpeg']
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'-F', u'https://player.vimeo.com/video/113927055?fun=0', u'--referer', u'https://videotextonline.com/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2021.12.17
[debug] Git HEAD: dd9aa74be
[debug] Python version 2.7.18 (CPython) - Linux-4.4.0-210-generic-i686-with-Ubuntu-16.04-xenial
[debug] exe versions: avconv 4.3, avprobe 4.3, ffmpeg 4.3, ffprobe 4.3
[debug] Proxy map: {}
[vimeo] 113927055: Downloading webpage
[vimeo] 113927055: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 113927055: Downloading akfire_interconnect_quic m3u8 information
[vimeo] 113927055: Downloading fastly_skyfire m3u8 information
[vimeo] 113927055: Downloading fastly_skyfire m3u8 information
[vimeo] 113927055: Downloading akfire_interconnect_quic MPD information
[vimeo] 113927055: Downloading akfire_interconnect_quic MPD information
[vimeo] 113927055: Downloading fastly_skyfire MPD information
[vimeo] 113927055: Downloading fastly_skyfire MPD information
[info] Available formats for 113927055:
format code                                          extension  resolution note
hls-akfire_interconnect_quic_sep-audio-medium-audio  mp4        audio only 
hls-fastly_skyfire_sep-audio-medium-audio            mp4        audio only 
dash-akfire_interconnect_quic_sep-audio-1fae5093     m4a        audio only DASH audio  114k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-fastly_skyfire_sep-audio-1fae5093               m4a        audio only DASH audio  114k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-akfire_interconnect_quic_sep-audio-6b5a4c2e     m4a        audio only DASH audio  162k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-fastly_skyfire_sep-audio-6b5a4c2e               m4a        audio only DASH audio  162k , m4a_dash container, mp4a.40.2 (48000Hz)
dash-akfire_interconnect_quic_sep-video-4a31d749     mp4        428x320    DASH video  351k , mp4_dash container, avc1.42C01E, video only
dash-fastly_skyfire_sep-video-4a31d749               mp4        428x320    DASH video  351k , mp4_dash container, avc1.42C01E, video only
hls-akfire_interconnect_quic_sep-411                 mp4        428x320     411k , avc1.42C01E, 29.97fps, video only
hls-fastly_skyfire_sep-411                           mp4        428x320     411k , avc1.42C01E, 29.97fps, video only
dash-akfire_interconnect_quic_sep-video-1fae5093     mp4        640x480    DASH video  717k , mp4_dash container, avc1.64001F, video only
dash-fastly_skyfire_sep-video-1fae5093               mp4        640x480    DASH video  717k , mp4_dash container, avc1.64001F, video only
hls-akfire_interconnect_quic_sep-692                 mp4        640x480     692k , avc1.64001F, 29.97fps, video only
hls-fastly_skyfire_sep-692                           mp4        640x480     692k , avc1.64001F, 29.97fps, video only
dash-akfire_interconnect_quic_sep-video-6b5a4c2e     mp4        720x540    DASH video 1194k , mp4_dash container, avc1.64001F, video only
dash-fastly_skyfire_sep-video-6b5a4c2e               mp4        720x540    DASH video 1194k , mp4_dash container, avc1.64001F, video only
hls-akfire_interconnect_quic_sep-1173                mp4        720x540    1173k , avc1.64001F, 29.97fps, video only
hls-fastly_skyfire_sep-1173                          mp4        720x540    1173k , avc1.64001F, 29.97fps, video only
dash-akfire_interconnect_quic-video-4a31d749         mp4        428x320    DASH video  465k , mp4_dash container, avc1.42C01E, mp4a.40.2 (48000Hz)
dash-fastly_skyfire-video-4a31d749                   mp4        428x320    DASH video  465k , mp4_dash container, avc1.42C01E, mp4a.40.2 (48000Hz)
hls-akfire_interconnect_quic-411                     mp4        428x320     411k , avc1.42C01E, 29.97fps, mp4a.40.2
hls-fastly_skyfire-411                               mp4        428x320     411k , avc1.42C01E, 29.97fps, mp4a.40.2
http-360p                                            mp4        428x320    30fps
dash-akfire_interconnect_quic-video-1fae5093         mp4        640x480    DASH video  831k , mp4_dash container, avc1.64001F, mp4a.40.2 (48000Hz)
dash-fastly_skyfire-video-1fae5093                   mp4        640x480    DASH video  831k , mp4_dash container, avc1.64001F, mp4a.40.2 (48000Hz)
hls-akfire_interconnect_quic-692                     mp4        640x480     692k , avc1.64001F, 29.97fps, mp4a.40.2
hls-fastly_skyfire-692                               mp4        640x480     692k , avc1.64001F, 29.97fps, mp4a.40.2
http-480p                                            mp4        640x480    30fps
dash-akfire_interconnect_quic-video-6b5a4c2e         mp4        720x540    DASH video 1355k , mp4_dash container, avc1.64001F, mp4a.40.2 (48000Hz)
dash-fastly_skyfire-video-6b5a4c2e                   mp4        720x540    DASH video 1355k , mp4_dash container, avc1.64001F, mp4a.40.2 (48000Hz)
hls-akfire_interconnect_quic-1221                    mp4        720x540    1221k , avc1.64001F, 29.97fps, mp4a.40.2
hls-fastly_skyfire-1221                              mp4        720x540    1221k , avc1.64001F, 29.97fps, mp4a.40.2
http-540p                                            mp4        720x540    30fps (best)
$

Get the downstream project to update yt-dl.

Duplicate of #31311.

Casper042 commented 1 year ago

Is there a Container version of the "Git Master"? I'd love to avoid setting up all the dependencies myself. Will happily give this alternate a try if so.

Thanks

dirkf commented 1 year ago

There aren't really any dependencies beyond the Python environment. Add a reasonably new ffmpeg and you're good to go, especially with a newer Python.

Casper042 commented 1 year ago

I ended up using MikeNye's docker as it seemed to have everything needed:

In case it helps anyone else, I dumped it all in a shell script that looks like the following:

#!/bin/bash

declare -a VIDEOS
VIDEOS=(113934772 113938782 113938787)
DOCKER="docker run \
    --rm -i \
    -e PGID=$(id -g) \
    -e PUID=$(id -u) \
    -v /Host/Output/Folder/:/workdir:rw \
    mikenye/youtube-dl"

rm -rf temp.sh
for VID in "${VIDEOS[@]}"
do
    CMD="${DOCKER} --referer 'https://videotextonline.com/' 'https://player.vimeo.com/video/${VID}?fun=0'"
    echo ${CMD} >>temp.sh
done

chmod +x temp.sh
echo RUN THIS TO DOWNLOAD:     ./temp.sh

The Videos array now only needs the ID of each Video to download The ECHO and temp.sh was because I couldn't get past some weirdness with the single and double quotes. Running it interactively would fail, but echo out to another script and then run that works fine.

Thanks for your guidance Dirk, it certainly led me down the right path. -Casper

PS: I only later discovered there is an option in YTDL to feed it an array/list of videos, so this might be the long way around, but it works for me and is Docker friendly.