iOS Application use youtube-dl

[noveleven]

Hi, i'm developing a iOS app to download videos. Now, python3 has been embedded in my iOS project, and i can call python function correctly use C method in '.m' file. I had downloaded youtube-dl-2016.03.14 source to my workspace, and i fetched youtube_dl directory to my project. When i used this code(below) to call youtube_dl module, script died with no error?

import youtube_dl
class MyLogger(object):
    def debug(self, msg):
        print("dubug:"+msg)
    def warning(self, msg):
        print("warning:"+msg)
    def error(self, msg):
        print("error:"+msg)
def download(prefix):
    otp = prefix+'/%(title)s.%(ext)s'
    print(otp)
    ydl_opts = {'outtmpl': otp,'logger': MyLogger(),}
    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        ydl.download(['http://www.nicovideo.jp/watch/sm28391399?cc_referrer=info_top'])`

Then, i traced script running, i found: script died or stopped, when execute "merger = FFmpegMergerPP(self)" in YoutubeDL.py.

[yan12125]

Could you run youtube-dl directly in a shell? (In a terminal app or SSH into your device) For example:

python3 youtube-dl/__main__.py 'http://www.nicovideo.jp/watch/sm28391399?cc_referrer=info_top'

If there's no python3 binary or a shell, please add the verbose flag to ydl_opts and post all the output.

import youtube_dl

class MyLogger(object):
    def debug(self, msg):
        print("dubug:"+msg)

    def warning(self, msg):
        print("warning:"+msg)

    def error(self, msg):
        print("error:"+msg)

def download(prefix):
    otp = prefix+'/%(title)s.%(ext)s'
    print(otp)
    ydl_opts = {
        'outtmpl': otp,
        'logger': MyLogger(),
        'verbose': True,
    }
    with youtube_dl.YoutubeDL(ydl_opts) as ydl:
        ydl.download(['http://www.nicovideo.jp/watch/sm28391399?cc_referrer=info_top'])

[noveleven]

@yan12125 打开了verbose，输出信息如下 (The output goes as following with the verbose flag) [debug] Encodings: locale US-ASCII, fs utf-8, out US-ASCII, pref US-ASCII [debug] youtube-dl version 2016.03.14 [debug] Python version 3.4.2 - Darwin-14.0.0-iPod5,1-32bit 然后就没有其他输出了。(No more outputs below it.)

关闭verbose时的输出如下(And without the verbose flag, outputs are) dubug:[niconico] sm28391399: Downloading webpage dubug:[niconico] sm28391399: Downloading video info page dubug:[niconico] sm28391399: Downloading webpage dubug:[niconico] sm28391399: Downloading flv info 然后就没有其他输出了。(No more outputs, either.)

Edit by @yan12125: Add English translations

[yan12125]

Error messages are printed to STDOUT. Could you catch the content of STDOUT?

Also, please use English so that other developers can understand the situation.

[yan12125]

Sorry, it should be STDERR but not STDOUT.

[noveleven]

@yan12125 , thanks for your patience, i got some error messages, such as:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/test2.py", line 19, in download
    ydl.download(['http://www.nicovideo.jp/watch/sm28391399?cc_referrer=info_top'])
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1739, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 682, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 729, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1364, in process_video_result
    merger = FFmpegMergerPP(self)
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/postprocessor/ffmpeg.py", line 50, in __init__
    self._determine_executables()
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/postprocessor/ffmpeg.py", line 113, in _determine_executables
    (p, get_exe_version(p, args=['-version'])) for p in programs)
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/postprocessor/ffmpeg.py", line 113, in <genexpr>
    (p, get_exe_version(p, args=['-version'])) for p in programs)
  File "/private/var/mobile/Containers/Bundle/Application/835059AA-EA47-4E56-8A0B-3AD92EE9A019/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/utils.py", line 1589, in get_exe_version
    stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate()
AttributeError: 'module' object has no attribute 'Popen'

[yan12125]

I guess you are using https://github.com/pybee/Python-iOS-template? Here's an excerpt from https://raw.githubusercontent.com/pybee/Python-iOS-support/master/patch/Python/Python.patch:

if sys.platform == 'ios':
    # iOS doesn't support subprocesses; however, some modules still import
    # subprocess, so it's helpful to retain the symbols in the namespace.
    __all__ = ["PIPE", "STDOUT", "DEVNULL", "CalledProcessError"]

And seems subprocess is not working on iOS without jailbreaking: http://stackoverflow.com/questions/4767853/running-subprocess-in-ios.

Without a functional subprocess module, at least the following functions does not work in youtube-dl:

1. Download HLS, RTMP, RTSP and MMS streams. RTSP and MMS are uncommon, while HLS and RTMP are everywhere. Most live streams use HLS, including YouTube live streams and Twitch.
2. Merge video and audio only files to a single file. YouTube serves such files for most HD streams. Facebook, TED and more sites serves such formats for some videos as well.
3. Using external downloaders to download media files, including curl, wget, axel, aria2 and httpie.
4. Embed thumbnails via ffmpeg or atomicparsley
5. Write extended attributes with xattr
6. Extract audio with -x, convert videos with --recode-video, embed subtitles with --embed-subs, add metadata to downloaded files with --add-metadata, fixup problematic files with --fixup and convert subtitles with --convert-subs. All of these functions requires calling external ffmpeg via subprocess.
7. Running custom commands with --exec

If all of the above are tolerable, working around incomplete subprocess requires lots of work. I don't have an iOS device so I can't test it. If anyone is interested in it, go ahead.

If you are willing to help, the first step is changing the following line in get_exe_version and check_executable functions of youtube_dl/utils.py from:

    except OSError:

to:

    except (OSError, AttributeError):

[noveleven]

Yes, i referred to the project to embed python, and i used his "Python-iOS-support" framework to develop my app. I has modified except OSError to except (OSError, AttributeError): in get_exe_version and check_executable function of youtube_dl/utils.py file, then the download feature of youtube-dl works fine in my app. But, i tried to download video from youtube site, and got these error messages:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/private/var/mobile/Containers/Bundle/Application/55F81DDA-C2D7-468F-8A39-635609ED58DC/py3objc.app/python/lib/python3.4/site-packages/test2.py", line 26, in download
    ydl.download([url])
  File "/private/var/mobile/Containers/Bundle/Application/55F81DDA-C2D7-468F-8A39-635609ED58DC/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1714, in download
  url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "/private/var/mobile/Containers/Bundle/Application/55F81DDA-C2D7-468F-8A39-635609ED58DC/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 682, in extract_info
    self.report_error(compat_str(e), e.format_traceback())
  File "/private/var/mobile/Containers/Bundle/Application/55F81DDA-C2D7-468F-8A39-635609ED58DC/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 543, in report_error
    self.trouble(error_message, tb)
  File "/private/var/mobile/Containers/Bundle/Application/55F81DDA-C2D7-468F-8A39-635609ED58DC/py3objc.app/python/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 513, in trouble
    raise DownloadError(message, exc_info)
youtube_dl.utils.DownloadError: ERROR: Unable to download webpage: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)> (caused by URLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)'),))

[yan12125]

There are several discussions about CERTIFICATE_VERIFY_FAILED. A possible solution on Mac OS X is at https://github.com/rg3/youtube-dl/issues/5317#issuecomment-89633861. I guess iOS is similar to Mac OS X. It that fails, a workaround can be nocheckcertificate:

    ydl_opts = {
        'outtmpl': otp,
        'logger': MyLogger(),
        'verbose': True,
        'nocheckcertificate': Truem
    }

NOTE: With this option, HTTPS is as insecure as plain HTTP.

[noveleven]

@yan12125 , thanks a lot, 'nocheckcertificate' option solved the problem!

[yan12125]

Note that nocheckcertificate is not a solution but a workaround, and it introduces unnecessary security holes.