search

ytdl-org / youtube-dl

Command-line program to download videos from YouTube.com and other video sites
http://ytdl-org.github.io/youtube-dl/
The Unlicense
131.67k stars 9.97k forks source link

`--proxy` not working properly . #29756

Closed other-half closed 3 years ago

other-half commented 3 years ago

Checklist

Verbose log

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', '--proxy', 'socks4://149.56.1.48:8181/', 'https://www.cbc.ca/player/play/1930912835522']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.8.10 (CPython) - Linux-5.11.0-25-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {'http': 'socks4://149.56.1.48:8181/', 'https': 'socks4://149.56.1.48:8181/'}
[ThePlatform] 1930912835522: Downloading SMIL data
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 475, in main
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 465, in _real_main
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2068, in download
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 847, in __extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 891, in process_ie_result
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 836, in __extract_info
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 534, in extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/theplatform.py", line 308, in _real_extract
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/theplatform.py", line 54, in _extract_theplatform_smil
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1993, in _parse_smil_formats
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 1472, in _is_valid_url
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 634, in _request_webpage
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2288, in urlopen
  File "/usr/lib/python3.8/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/lib/python3.8/urllib/request.py", line 542, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2735, in https_open
  File "/usr/lib/python3.8/urllib/request.py", line 1354, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/usr/lib/python3.8/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.8/http/client.py", line 1007, in _send_output
    self.send(msg)
  File "/usr/lib/python3.8/http/client.py", line 947, in send
    self.connect()
  File "/usr/local/bin/youtube-dl/youtube_dl/utils.py", line 2703, in connect
  File "/usr/local/bin/youtube-dl/youtube_dl/socks.py", line 270, in connect
  File "/usr/local/bin/youtube-dl/youtube_dl/socks.py", line 266, in _make_proxy
  File "/usr/local/bin/youtube-dl/youtube_dl/socks.py", line 167, in _setup_socks4
  File "/usr/local/bin/youtube-dl/youtube_dl/socks.py", line 126, in recvall
EOFError: 8 bytes missing

Description

I was trying to see all resolutions of a video from cbc.ca using --proxy, but it is showing the above message . But for another video from the same website (cbc.ca) and using the same proxy I was able to see resolutions list, still while trying to download it (by removing -F) it showed ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command. ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers. After adding --hls-prefer-native it showed the above verbose output. Can anyone help me resolve it ?

ghost commented 3 years ago

Did you confirm the video can be played normally in a browser through that socks4://149.56.1.48:8181/ proxy?

I simply think you need to find other proxy servers. It seems to work but not so fast, sometimes get timeout.

$ youtube-dl https://www.cbc.ca/player/play/1930912835522 --ignore-config --hls-prefer-native --proxy socks4://149.56.1.48:8181/
[ThePlatform] 1930912835522: Downloading SMIL data
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Downloading JSON metadata
[download] Destination: Olympic flame is extinguished to close Tokyo 2020-1930912835522.mp4
[download]  19.5% of 83.26MiB at 639.79KiB/s ETA 01:47^C
Vangelis66 commented 3 years ago

In the case of cbc.ca, you ONLY need to proxy PlaylistAPI (ThePlatform) requests, as the stream CDN itself does not geo-fence; you may hence use a working Canadian proxy (with moderate available bandwidth) as the value of the --geo-verification-proxy switch, stream will come down DIRECT 😜 

youtube-dl --geo-verification-proxy "http://proxyhost:proxyport" -F "https://www.cbc.ca/player/play/1930912835522"

[ThePlatform] 1930912835522: Downloading SMIL data
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Downloading JSON metadata
[info] Available formats for 1930912835522:
format code  extension  resolution note
http-321     mp4        320x180     321k
http-466     mp4        320x180     466k
http-832     mp4        640x360     832k
http-1232    mp4        852x480    1232k
http-1932    mp4        852x480    1932k
http-2632    mp4        1280x720   2632k
http-7132    mp4        1920x1080  7132k  (best)

and... 

youtube-dl --geo-verification-proxy "http://proxyhost:proxyport" "https://www.cbc.ca/player/play/1930912835522"

[ThePlatform] 1930912835522: Downloading SMIL data
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Checking video URL
[ThePlatform] 1930912835522: Downloading JSON metadata
[download] Destination: Olympic flame is extinguished to close Tokyo 2020-193091
2835522.mp4
[download]   2.4% of 83.26MiB at  1.20MiB/s ETA 01:07
other-half commented 3 years ago

Thank you, It worked for some videos like the one I said earlier, I was able to download it with a proxy (socks4://69.70.245.146:4153). But when I tried to download another video (https://www.cbc.ca/player/play/1922081347533) with the same proxy it showed error message "HTTP error 403 Forbidden" though I was able to see all the resolution available by adding -F. I also tried playing the video using the same proxy from a browser and it played fine.

$ youtube-dl -F -v --geo-verification-proxy socks4://69.70.245.146:4153  https://www.cbc.ca/player/play/1922081347533
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-F', '-v', '--geo-verification-proxy', 'socks4://69.70.245.146:4153', 'https://www.cbc.ca/player/play/1922081347533']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.8.10 (CPython) - Linux-5.11.0-25-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[ThePlatform] 1922081347533: Downloading SMIL data
[ThePlatform] 1922081347533: Downloading m3u8 information
[ThePlatform] 1922081347533: Downloading JSON metadata
[info] Available formats for 1922081347533:
format code  extension  resolution note
hls-422      mp4        416x234     422k , avc1.4d400d, 30.0fps, mp4a.40.2
hls-580      mp4        480x270     580k , avc1.4d4015, 30.0fps, mp4a.40.2
hls-910      mp4        640x360     910k , avc1.77.30, 30.0fps, mp4a.40.2
hls-1460     mp4        768x432    1460k , avc1.77.30, 30.0fps, mp4a.40.2
hls-2340     mp4        960x540    2340k , avc1.4d401f, 30.0fps, mp4a.40.2
hls-3440     mp4        1280x720   3440k , avc1.4d401f, 30.0fps, mp4a.40.2 (best)

and when I tried to download it, the output was this

(without -v)

$ youtube-dl --geo-verification-proxy socks4://69.70.245.146:4153  https://www.cbc.ca/player/play/1922081347533
[ThePlatform] 1922081347533: Downloading SMIL data
[ThePlatform] 1922081347533: Downloading m3u8 information
[ThePlatform] 1922081347533: Downloading JSON metadata
[download] Destination: Equestrian - Jumping Team - Final-1922081347533.mp4
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[https @ 0x55c1109ff780] HTTP error 403 Forbidden
https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1628965382~acl=%2f*~id=c3629265-79cc-4c0a-b734-b81f2fd50227~data=hdntl~hmac=2893ecebb182f438be2cbf69798ff22ff4eebccc2f30eeba6f38786a40381e80/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAAAGChhhAAAAAJ8h%2f%2fvh0IDwWPdC26kdNeVRg1ydgH2HCKiUGSWI4JgNMaOpvCDDkcQiP4rxBPevIfPZiWcs2v3y&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816: Server returned 403 Forbidden (access denied)

ERROR: ffmpeg exited with code 1

(with -v)

$ youtube-dl -v --geo-verification-proxy socks4://69.70.245.146:4153  https://www.cbc.ca/player/play/1922081347533
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--geo-verification-proxy', 'socks4://69.70.245.146:4153', 'https://www.cbc.ca/player/play/1922081347533']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.8.10 (CPython) - Linux-5.11.0-25-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {}
[ThePlatform] 1922081347533: Downloading SMIL data
[ThePlatform] 1922081347533: Downloading m3u8 information
[ThePlatform] 1922081347533: Downloading JSON metadata
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1628965288~acl=%2f*~id=fe6f38ff-1ef2-4104-bf38-01cce7a70bb4~data=hdntl~hmac=c74dc672cc24912b34edf91f385661e092674f55eab0517d0d424e0e66ed5c49/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAACoCRhhAAAAADdVFxsnXdzKnBxewM1oiZjvIqfDFLoGlRHHw9xDF5Pb4YigBPgBq%2fSq5ZjOSSDY5Y9wRhhTvWkR&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816'
[download] Destination: Equestrian - Jumping Team - Final-1922081347533.mp4
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.106 Safari/537.36
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i 'https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1628965288~acl=%2f*~id=fe6f38ff-1ef2-4104-bf38-01cce7a70bb4~data=hdntl~hmac=c74dc672cc24912b34edf91f385661e092674f55eab0517d0d424e0e66ed5c49/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAACoCRhhAAAAADdVFxsnXdzKnBxewM1oiZjvIqfDFLoGlRHHw9xDF5Pb4YigBPgBq%2fSq5ZjOSSDY5Y9wRhhTvWkR&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816' -c copy -f mp4 'file:Equestrian - Jumping Team - Final-1922081347533.mp4.part'
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[tcp @ 0x55cd3f86af00] Starting connection attempt to 23.213.1.41 port 443
[tcp @ 0x55cd3f86af00] Successfully connected to 23.213.1.41 port 443
[https @ 0x55cd3f8687c0] HTTP error 403 Forbidden
https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1628965288~acl=%2f*~id=fe6f38ff-1ef2-4104-bf38-01cce7a70bb4~data=hdntl~hmac=c74dc672cc24912b34edf91f385661e092674f55eab0517d0d424e0e66ed5c49/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAACoCRhhAAAAADdVFxsnXdzKnBxewM1oiZjvIqfDFLoGlRHHw9xDF5Pb4YigBPgBq%2fSq5ZjOSSDY5Y9wRhhTvWkR&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816: Server returned 403 Forbidden (access denied)

ERROR: ffmpeg exited with code 1
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 475, in main
    _real_main(argv)
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2068, in download
    res = self.extract_info(
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 847, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 918, in process_ie_result
    return self.process_ie_result(
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 881, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1692, in process_video_result
    self.process_info(new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1976, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/external.py", line 63, in real_download
    self.report_error('%s exited with code %d' % (
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 628, in report_error
    self.trouble(error_message, tb)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 590, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

Can someone help me with this problem?

Thanks.

ghost commented 3 years ago

--hls-prefer-native --proxy socks4://69.70.245.146:4153

other-half commented 3 years ago

--hls-prefer-native --proxy socks4://69.70.245.146:4153 too didn't work. I was able to see the available resolution but wasn't able to download it. The output was same as before. This time I used a different proxy (socks4://67.213.111.124:5678) and also I was able to play the video using this proxy from a browser.

without -v

$ youtube-dl --proxy socks4://67.213.111.124:5678  https://www.cbc.ca/player/play/1922081347533
[ThePlatform] 1922081347533: Downloading SMIL data
[ThePlatform] 1922081347533: Downloading m3u8 information
[ThePlatform] 1922081347533: Downloading JSON metadata
[download] Destination: Equestrian - Jumping Team - Final-1922081347533.mp4
WARNING: ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command.
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[https @ 0x5559538e8780] HTTP error 403 Forbidden
https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1629014232~acl=%2f*~id=573b387b-6d03-430e-8df7-695a5daa054b~data=hdntl~hmac=ce0f12d074b7adeaed669626ff43eb7b64cdd6566ed97ea2f09137d43928aa0e/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAADYyBhhAAAAAK3KzjciyYVZ2+5289LvA7WL3LG56Nmpw+1liiz9+ubkf+IBuEVMduFnLPEtN4CgjAY3RoN4Jl3o&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816: Server returned 403 Forbidden (access denied)

ERROR: ffmpeg exited with code 1

with -v

$ youtube-dl -v --proxy socks4://67.213.111.124:5678  https://www.cbc.ca/player/play/1922081347533
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v', '--proxy', 'socks4://67.213.111.124:5678', 'https://www.cbc.ca/player/play/1922081347533']
[debug] Encodings: locale UTF-8, fs utf-8, out utf-8, pref UTF-8
[debug] youtube-dl version 2021.06.06
[debug] Python version 3.8.10 (CPython) - Linux-5.11.0-25-generic-x86_64-with-glibc2.29
[debug] exe versions: ffmpeg 4.2.4, ffprobe 4.2.4, rtmpdump 2.4
[debug] Proxy map: {'http': 'socks4://67.213.111.124:5678', 'https': 'socks4://67.213.111.124:5678'}
[ThePlatform] 1922081347533: Downloading SMIL data
[ThePlatform] 1922081347533: Downloading m3u8 information
[ThePlatform] 1922081347533: Downloading JSON metadata
[debug] Default format spec: bestvideo+bestaudio/best
[debug] Invoking downloader on 'https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1629014334~acl=%2f*~id=e14deb49-2672-4930-bb6f-3413518099a3~data=hdntl~hmac=99753bae28e4e5a42a23dcd99b1e91e51871c278255b8701d555abb1e224fae5/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAAA+yRhhAAAAABVyT7g8NgBaOXkuO3gyDC9jrqR5FChbswfgqLtZ7FCKd9wO+gPZvfAtTjcVOCT5HGP9w2hl6f0j&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816'
[download] Destination: Equestrian - Jumping Team - Final-1922081347533.mp4
WARNING: ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command.
[debug] ffmpeg command line: ffmpeg -y -loglevel verbose -headers 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3704.1 Safari/537.36
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i 'https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1629014334~acl=%2f*~id=e14deb49-2672-4930-bb6f-3413518099a3~data=hdntl~hmac=99753bae28e4e5a42a23dcd99b1e91e51871c278255b8701d555abb1e224fae5/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAAA+yRhhAAAAABVyT7g8NgBaOXkuO3gyDC9jrqR5FChbswfgqLtZ7FCKd9wO+gPZvfAtTjcVOCT5HGP9w2hl6f0j&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816' -c copy -f mp4 'file:Equestrian - Jumping Team - Final-1922081347533.mp4.part'
ffmpeg version 4.2.4-1ubuntu0.1 Copyright (c) 2000-2020 the FFmpeg developers
  built with gcc 9 (Ubuntu 9.3.0-10ubuntu2)
  configuration: --prefix=/usr --extra-version=1ubuntu0.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-avresample --disable-filter=resample --enable-avisynth --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-nvenc --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[tcp @ 0x55d9282e5f00] Starting connection attempt to 23.45.91.237 port 443
[tcp @ 0x55d9282e5f00] Successfully connected to 23.45.91.237 port 443
[https @ 0x55d9282e37c0] HTTP error 403 Forbidden
https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1629014334~acl=%2f*~id=e14deb49-2672-4930-bb6f-3413518099a3~data=hdntl~hmac=99753bae28e4e5a42a23dcd99b1e91e51871c278255b8701d555abb1e224fae5/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAAA+yRhhAAAAABVyT7g8NgBaOXkuO3gyDC9jrqR5FChbswfgqLtZ7FCKd9wO+gPZvfAtTjcVOCT5HGP9w2hl6f0j&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816: Server returned 403 Forbidden (access denied)

ERROR: ffmpeg exited with code 1
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/bin/youtube-dl/__main__.py", line 19, in <module>
    youtube_dl.main()
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 475, in main
    _real_main(argv)
  File "/usr/local/bin/youtube-dl/youtube_dl/__init__.py", line 465, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 2068, in download
    res = self.extract_info(
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 808, in extract_info
    return self.__extract_info(url, ie, download, extra_info, process)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 815, in wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 847, in __extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 918, in process_ie_result
    return self.process_ie_result(
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 881, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1692, in process_video_result
    self.process_info(new_info)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1976, in process_info
    success = dl(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1915, in dl
    return fd.download(name, info)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/external.py", line 63, in real_download
    self.report_error('%s exited with code %d' % (
  File "/usr/local/bin/youtube-dl/youtube_dl/downloader/common.py", line 165, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 628, in report_error
    self.trouble(error_message, tb)
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 590, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())
ghost commented 3 years ago

Never works with ffmpeg because it doesn't support socks proxy.

Vangelis66 commented 3 years ago

@other-half

I wrote:

In the case of cbc.ca, you ONLY need to proxy PlaylistAPI (ThePlatform) requests, as the stream CDN itself does not geo-fence

It would appear my initial assessment is only partially true :wink: (but I recall it used to be like I said in the past...); with regards to geo-blocking, www.cbc.ca videos seem to fall under two groups:

  1. Those (usually short clips/excerpts of full shows) for which only the playlist API is geo-fenced, not the stream itself; as I wrote above, using only the --geo-verification-proxy option would suffice for a fetch...
  2. Those, the bulk of the www.cbc.ca offerings, i.e. full length programs/shows/events, for which both the Playlist API and media stream are geo-fenced, i.e. reserved only for whitelisted Canadian IPs.

In that latter case, if you care to inspect more closely your posted logs, you'll find that the master HLS playlist URI for https://www.cbc.ca/player/play/1922081347533, i.e. 

https://cbcrcolympic-cbc.akamaized.net/hls/live/2025682/geo_allow_ca/yv8y7e53v/hdntl=exp=1629014232~acl=%2f*~id=573b387b-6d03-430e-8df7-695a5daa054b~data=hdntl~hmac=ce0f12d074b7adeaed669626ff43eb7b64cdd6566ed97ea2f09137d43928aa0e/master_3000-archive.m3u8?aka_me_session_id=AAAAAAAAAADYyBhhAAAAAK3KzjciyYVZ2+5289LvA7WL3LG56Nmpw+1liiz9+ubkf+IBuEVMduFnLPEtN4CgjAY3RoN4Jl3o&aka_media_format_type=hls&startTime=1628330097&endTime=1628339816

contains the geo_allow_ca fragment, tell-tale sign it's CA-only; for these types of video, you'll have to use the --proxy option; are we clear thus far?

About the use of the --hls-prefer-native flag:

I am on Windows myself, sadly I know little about how things work under Ubuntu, plus it's not clear how you have configured your youtube-dl installation there...

On Windows, when using the officially provided youtube-dl.exe binary OR have a different setup but one that includes either of the pyCrypto or pyCryptodome Python modules, the HLS native python downloader kicks-in automatically for www.cbc.ca downloads (as can be seen in my logs), I don't have to explicitly issue the --hls-prefer-native flag... [ Important EDIT: This seems to be the case ONLY when a HTTP/HTTPS proxy is the value of the --proxy option; when a SOCKS proxy is used (OP's case), yt-dl defaults to FFmpeg downloader (with WARNING: ffmpeg does not support SOCKS proxies. Downloading is likely to fail. Consider adding --hls-prefer-native to your command.) 😞 ; maybe this is worth filing a bug report for? ]

It appears that under your setup, ffmpeg is invoked as the preferred HLS downloader, but, as pointed out in your many logs, FFmpeg does not support SOCKS proxies (only HTTP[S]); unfortunately, SOCKS is the type of proxy you insist on using and, as pointed out by @kikuyan, the value of the --proxy option (a socks4 proxy) CAN'T BE PASSED ON TO/USED BY ffmpeg, ffmpeg tries to fetch the stream DIRECT (via your non-CA IP) and "gets the door shut in its face" [Server returned 403 Forbidden (access denied)] ...

Thus, you always have to specify --hls-prefer-native when trying to fetch a fully-blocked stream with a SOCKS proxy; is it clear?

In closing, let me touch another aspect of your otherwise invalid issue: If you are in the habit of fetching media content normally blocked in your physical location, you should consider switching to something MORE RELIABLE than "free" proxies found on the net; those are usually compromised misconfigured systems, whose "lifespan/available bandwidth/connection speed/ping/etc are not guaranteed in ANY way... It is NOT a fault of youtube-dl's code if a "free" proxy you are employing to fetch content displays erratic performance, this isn't a genuine yt-dl bug (nor have you presented us with evidence it is), so I kindly ask you to close this issue. Explore one of the many paid VPN solutions (some offer free plans, like Windscribe)/SSH tunnels/HTTPS proxies to reliably fetch your Canadian content...

PS: While trying successfully a proxy in a browser is indeed concrete proof it hasn't been blacklisted by the media provider, often times the test isn't conclusive for yt-dl fetch purposes; the desktop browser might be offered different types of streams than the actual yt-dl IE/plugin (e.g. MPEG-DASH in browser, while HLS in yt-dl...); take note of this fact, too...

Kind regards

other-half commented 3 years ago

Thank you.