ffmpeg/avconv does not support SOCKS proxies

scfc commented 8 years ago

Make sure you are using the latest version: run `youtube-dl --version` and ensure your version is 2016.08.12. If it's not read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2016.08.12
Before submitting an issue make sure you have:
[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones
What is the purpose of your issue?
[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:

Add -v flag to your command line you run youtube-dl with, copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):

[tim@passepartout ~]$ youtube-dl -v --proxy socks5://127.0.0.1:11111/ http://www.cwtv.com/shows/penn-teller-fool-us/you-dirty-rathead/?play=0c8763f4-6dfc-42c2-86af-02e9fd078334
[debug] System config: ['--prefer-free-formats']
[debug] User config: []
[debug] Command-line args: ['-v', '--proxy', 'socks5://127.0.0.1:11111/', 'http://www.cwtv.com/shows/penn-teller-fool-us/you-dirty-rathead/?play=0c8763f4-6dfc-42c2-86af-02e9fd078334']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.07.17
[debug] Python version 3.4.3 - Linux-4.6.5-200.fc23.x86_64-x86_64-with-fedora-23-Twenty_Three
[debug] exe versions: ffmpeg 2.8.7, ffprobe 2.8.7, rtmpdump 2.4
[debug] Proxy map: {'http': 'socks5://127.0.0.1:11111/', 'https': 'socks5://127.0.0.1:11111/'}
[CWTV] 0c8763f4-6dfc-42c2-86af-02e9fd078334: Downloading JSON metadata
[CWTV] 0c8763f4-6dfc-42c2-86af-02e9fd078334: Downloading m3u8 information
[debug] Invoking downloader on 'http://hlsioscwtv.warnerbros.com/hls/2016/06/28/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps.m3u8'
[download] Destination: You Dirty Rathead!-0c8763f4-6dfc-42c2-86af-02e9fd078334.mp4
[debug] ffmpeg command line: ffmpeg -y -headers 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
' -i http://hlsioscwtv.warnerbros.com/hls/2016/06/28/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps.m3u8 -c copy -f mp4 -bsf:a aac_adtstoasc 'file:You Dirty Rathead!-0c8763f4-6dfc-42c2-86af-02e9fd078334.mp4.part'
ffmpeg version 2.8.7 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.1 (GCC) 20160406 (Red Hat 5.3.1-6)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -m64 -mtune=generic' --enable-bzlib --disable-crystalhd --enable-frei0r --enable-gnutls --enable-ladspa --enable-libass --enable-libcdio --enable-libdc1394 --disable-indev=jack --enable-libfreetype --enable-libgsm --enable-libmp3lame --enable-openal --enable-libopencv --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libvorbis --enable-libv4l2 --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-x11grab --enable-avfilter --enable-avresample --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-runtime-cpudetect
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
[http @ 0x122f2c0] HTTP error 403 Forbidden
http://hlsioscwtv.warnerbros.com/hls/2016/06/28/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps/PennandTeller-305-YouDirtyRathead-P301-CW-Stereo_b2f7faf81_2100kbps.m3u8: Server returned 403 Forbidden (access denied)

ERROR: ffmpeg exited with code 1
  File "/usr/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2016.7.17', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3.4/site-packages/youtube_dl/__init__.py", line 422, in main
    _real_main(argv)
  File "/usr/lib/python3.4/site-packages/youtube_dl/__init__.py", line 412, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1775, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 693, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 739, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1421, in process_video_result
    self.process_info(new_info)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1683, in process_info
    success = dl(filename, info_dict)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1625, in dl
    return fd.download(name, info)
  File "/usr/lib/python3.4/site-packages/youtube_dl/downloader/common.py", line 350, in download
    return self.real_download(filename, info_dict)
  File "/usr/lib/python3.4/site-packages/youtube_dl/downloader/external.py", line 43, in real_download
    self.get_basename(), retval))
  File "/usr/lib/python3.4/site-packages/youtube_dl/downloader/common.py", line 161, in report_error
    self.ydl.report_error(*args, **kargs)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 556, in report_error
    self.trouble(error_message, tb)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 518, in trouble
    tb_data = traceback.format_list(traceback.extract_stack())

[tim@passepartout ~]$

Description of your issue, suggested solution and other information

If I run youtube-dl on a host in the US to download geo-restricted content like http://www.cwtv.com/shows/penn-teller-fool-us/you-dirty-rathead/?play=0c8763f4-6dfc-42c2-86af-02e9fd078334, it works fine. If I set up a SOCKS proxy and use --proxy socks5://127.0.0.1:11111/, it fails. I assume this is because youtube-dl calls ffmpeg, but does not instruct ffmpeg to use the proxy.

This is confusing. Either youtube-dl should pass proxy information to subcommands, or, if youtube-dl was called with the --proxy option and a subcommand does not support using a SOCKS proxy, youtube-dl should abort with an informative error message.

yan12125 commented 8 years ago

if youtube-dl was called with the --proxy option and a subcommand does not support using a SOCKS proxy, youtube-dl should abort with an informative error message.

Seems ffmpeg does not support SOCKS5 at all, is it? It's a good idea.

yan12125 commented 8 years ago

By the way, could you try to install python3-crypto and then run youtube-dl with --hls-prefer-native?

yan12125 commented 8 years ago

Requested to ffmpeg at http://trac.ffmpeg.org/ticket/5776.

scfc commented 8 years ago

I successfully tried --hls-prefer-native, and for the URL mentioned I actually prefer it :-) because the resulting file shrinks from 1.5 GByte to 598 MByte without any noticeable reduction in quality.

yan12125 commented 8 years ago

I've added a warning if SOCKS is used with ffmpeg. I didn't raise an error as some SOCKS proxies may also work as HTTP proxies.

aeroxy commented 6 years ago

Hi, for macOS users what should we do to enable --hls-prefer-native flag? brew install python3-crypto does not install python3-crypto (no package found).

yan12125 commented 6 years ago

I remember Homebrew doesn't provide Python packages. Try to install it with pip; e.g. pip install crypto --user

remitamine commented 6 years ago

just a note, pycryptodome might be faster in the decryption as it does support AES-NI(not included in the latest stable version of pycrypto).

aeroxy commented 6 years ago

@yan12125 maybe you mean pip3 install crypto --user? Which python does youtube-dl uses by default? Python or Python3?

aeroxy commented 6 years ago

@remitamine so I guess I can use pip3 install cryptodom --user?

yan12125 commented 6 years ago

Which python does youtube-dl uses by default? Python or Python3?

That depends. You can check the version from youtube-dl -v

pip3 install cryptodome --user should work, too.

aeroxy commented 6 years ago

@yan12125 looks like it uses Python version 2.7.10 - Darwin-17.2.0-x86_64-i386-64bit, so the follow up question is, can I change the default Python and if not is there a cryptodome for python 2?

aeroxy commented 6 years ago

Also it looks like when I uses pip I would just get pip: command not found...

aeroxy commented 6 years ago

I do have pip3 however...not sure wtf is going on with macOS these days.

dwarf007 commented 4 years ago

For me the following worked: pip install pycryptodome

Don't forget to source you ~/.bashrc or ~/.zshrc afterwards..

ytdl-org / youtube-dl