Add aria2c as external downloader

[JunkFood02]

• See https://github.com/JunkFood02/Seal/issues/148

[xibr]

In my opinion let's give it more testing, and see the issues that need to be solved, I also noticed that there is a problem with the text progress bar sometimes it is not updated.

I'll create a pull request maybe tomorrow until we test it well then let's decide what we're going to do.

[xibr]

It works fine but we have a problem with the text progress bar.

[JunkFood02]

It works fine but we have a problem with the text progress bar.

Reference

I'll be looking into it

[JunkFood02]

Did some testing. The console output of aria2c could be parsed by the regex below: \[#\w{6}.*\((\d*\.*\d+)%\).*?((\d+)m)*((\d+)s)*]

The result would be matcher.group(1), matcher.group(3) (nullable), matcher.group(5) (nullable)

Test input:

"[#2089b0 400.0KiB/33.2MiB(1.2%) CN:1 DL:115.7KiB ETA:1m51s]",
"[#2089b0 400.0KiB/33.2MiB(100.0%) CN:1 DL:115.7KiB ETA:51s]",
"[#2089b0 400.0KiB/33.2MiB(0%) CN:1 DL:115.7KiB ETA:51s]",
"[#2089b0 400.0KiB/33.2MiB(12.3%) CN:1 DL:115.7KiB ETA:00s]",
"[#6e26e9 3.3MiB/4.3MiB(77%) CN:1 DL:1.7MiB]",
"[#6e26e9 3.4MiB/4.3MiB(78%) CN:1 DL:1.0MiB]",
"[#6e26e9 3.4MiB/4.3MiB(79%) CN:1 DL:813KiB ETA:1s]",
"[#6e26e9 3.4MiB/4.3MiB(80%) CN:1 DL:639KiB ETA:1s]",
"[#6e26e9 3.5MiB/4.3MiB(81%) CN:1 DL:534KiB ETA:1s]",

Test output:

1.2 1m 1 51s 51
100.0 null null 51s 51
0 null null 51s 51
12.3 null null 00s 00
77 null null null null
78 null null null null
79 null null 1s 1
80 null null 1s 1
81 null null 1s 1

The problem now is how to decide the pattern for parsing, and how to expose the API to the outer caller

[JunkFood02]

After doing some tests with the library, it turns out that the yt-dlp process will be blocked by the aria2c process when downloading, along with the stdout stream. As a result, we cannot directly read the output of the aria2c from the stdout of yt-dlp.

[xibr]

After several attempts --summary-interval=1 seems to solve the problem, please check.

[xibr]

After doing some tests with the library, it turns out that the yt-dlp process will be blocked by the aria2c process when downloading, along with the stdout stream. As a result, we cannot directly read the output of the aria2c from the stdout of yt-dlp.

At first this is what i thought but i ran aria2c directly to test it without yt-dlp but there was a problem the progress bar is not showing.

[JunkFood02]

It works like a charm 😉

Screenshot


[xibr]

If there is no problem while testing this, we will need to add this patch for yt-dlp to make it work on youtubedl-android or if there is a better idea to use libaria2c.so

from https://github.com/JunkFood02/Seal/issues/148#issuecomment-1224440508

diff --git a/yt_dlp/downloader/external.py b/yt_dlp/downloader/external.py
index 9859a7b33..635a8d867 100644
--- a/yt_dlp/downloader/external.py
+++ b/yt_dlp/downloader/external.py
@@ -239,7 +239,7 @@ def _make_cmd(self, tmpfilename, info_dict):
         return cmd

-class Aria2cFD(ExternalFD):
+class Libaria2cFD(ExternalFD):
     AVAILABLE_OPT = '-v'
     SUPPORTED_PROTOCOLS = ('http', 'https', 'ftp', 'ftps', 'dash_frag_urls', 'm3u8_frag_urls')

This is how we will use it https://github.com/yausername/youtubedl-android/blob/fc70f3a1120a52e0b8094139257545fc7e241c2b/app/src/main/java/com/yausername/youtubedl_android_example/DownloadingExampleActivity.java#L131-L132

[pukkandan]

About the progress bar - we have plans to parse the progress in yt-dlp itself https://github.com/yt-dlp/yt-dlp/pull/3724. However, I can't give an ETA on this

[xibr]

About the progress bar - we have plans to parse the progress in yt-dlp itself yt-dlp/yt-dlp#3724.

I've tested yt-dlp/yt-dlp#3724 on youtubedl-android with the patch applied to run libaria2c.so but it seems I'm having a problem with No usable temporary directory found in ['/data/data/com.termux/files/usr/tmp', '/'] for now this problem can be solved by replacing /data/data/com.termux/files/usr/tmp with /storage/emulated/0/tmp in the tempfile.py in libpython.zip.so (May not work on Android 11+ version).

--verbose --external-downloader libaria2c.so -o /sdcard/Download/youtubedl-android/%(title)s.%(ext)s https://www.youtube.com/watch?v=dQw4w9WgXcQ

[debug] Command-line config: ['--verbose', '--external-downloader', 'libaria2c.so', '-o', '/sdcard/Download/youtubedl-android/%(title)s.%(ext)s', 'https://www.youtube.com/watch?v=dQw4w9WgXcQ', '--no-cache-dir', '--ffmpeg-location', '/data/app/com.yausername.youtubedl_android_example-VExKrjvp4wYvnW204QLXJw==/lib/arm64/libffmpeg.bin.so']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8 (No ANSI), error utf-8 (No ANSI), screen utf-8 (No ANSI)
[debug] yt-dlp version 2022.08.19.1 [48c88e088] (zip)
[debug] Python 3.8.0 (CPython 64bit) - Linux-4.4.111-21427293-aarch64-with-libc (libc)
[debug] Checking exe version: /data/app/com.yausername.youtubedl_android_example-VExKrjvp4wYvnW204QLXJw==/lib/arm64/libffmpeg.bin.so -bsfs
[debug] Checking exe version: /data/app/com.yausername.youtubedl_android_example-VExKrjvp4wYvnW204QLXJw==/lib/arm64/libffprobe.bin.so -bsfs
[debug] exe versions: ffmpeg 4.2.1, ffprobe 4.2.1
[debug] Optional libraries: mutagen-1.45.1, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1663 extractors
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=dQw4w9WgXcQ
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Default format spec: bestvideo*+bestaudio/best
[debug] Invoking libaria2c downloader on "https://rr3---sn-25ge7nse.googlevideo.com/videoplayback?expire=1661923918&ei=7p0OY66cJKzXxN8PiKeA2Ag&ip=212.47.228.201&id=o-ANTSXWRuHglCvHe45KCaxpMDuRifhe-HXBxUlpWIQ0rr&itag=137&source=youtube&requiressl=yes&mh=7c&mm=31%2C26&mn=sn-25ge7nse%2Csn-h5qzen7d&ms=au%2Conr&mv=m&mvi=3&pl=22&initcwndbps=80000&spc=lT-KhhGYDDvasS01khgY1uGrzm6GzFc&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=94013124&dur=212.040&lmt=1651673368972926&mt=1661901439&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&rbqsm=fr&txp=4535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgZKbQI5VQcCMJTMwiw1LqKcFeB14Wye8K8TbRUsjieAMCIFWemr6BXf72Ah1zKkjV-C90iDIYGeeumiu-quLoiLNV&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALZzMy-7b4WxxHD85s69rn1AAQifsVlDyXD2Eqa5PWs8AiEA9P2kV3RLYKkBd-sFGWQRuvmkg2DYm9Pa0wDgRe8hdis%3D"
[debug] libaria2c.so command line: libaria2c.so -c --console-log-level=warn --summary-interval=0 --download-result=hide --http-accept-gzip=true --file-allocation=none -x16 -j16 -s16 --min-split-size 1M --header 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36' --header 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' --header 'Accept-Language: en-us,en;q=0.5' --header 'Sec-Fetch-Mode: navigate' --check-certificate=true --remote-time=true --show-console-readout=true --enable-rpc --rpc-listen-port=40700 --dir /sdcard/Download/youtubedl-android/ --out './Rick Astley - Never Gonna Give You Up (Official Music Video).f137.mp4.part' --auto-file-renaming=false -- 'https://rr3---sn-25ge7nse.googlevideo.com/videoplayback?expire=1661923918&ei=7p0OY66cJKzXxN8PiKeA2Ag&ip=212.47.228.201&id=o-ANTSXWRuHglCvHe45KCaxpMDuRifhe-HXBxUlpWIQ0rr&itag=137&source=youtube&requiressl=yes&mh=7c&mm=31%2C26&mn=sn-25ge7nse%2Csn-h5qzen7d&ms=au%2Conr&mv=m&mvi=3&pl=22&initcwndbps=80000&spc=lT-KhhGYDDvasS01khgY1uGrzm6GzFc&vprv=1&svpuc=1&mime=video%2Fmp4&gir=yes&clen=94013124&dur=212.040&lmt=1651673368972926&mt=1661901439&fvip=4&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&rbqsm=fr&txp=4535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgZKbQI5VQcCMJTMwiw1LqKcFeB14Wye8K8TbRUsjieAMCIFWemr6BXf72Ah1zKkjV-C90iDIYGeeumiu-quLoiLNV&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhALZzMy-7b4WxxHD85s69rn1AAQifsVlDyXD2Eqa5PWs8AiEA9P2kV3RLYKkBd-sFGWQRuvmkg2DYm9Pa0wDgRe8hdis%3D'
ERROR: unable to download video data: [Errno 2] No usable temporary directory found in ['/data/data/com.termux/files/usr/tmp', '/']
Traceback (most recent call last):
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/YoutubeDL.py", line 3146, in process_info
    partial_success, real_download = self.dl(fname, new_info)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/YoutubeDL.py", line 2888, in dl
    return fd.download(name, new_info, subtitle)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/common.py", line 442, in download
    ret = self.real_download(filename, info_dict)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/external.py", line 48, in real_download
    retval = self._call_downloader(tmpfilename, info_dict)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/external.py", line 321, in _call_downloader
    return super()._call_downloader(tmpfilename, info_dict)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/external.py", line 135, in _call_downloader
    _, stderr, returncode = self._call_process(cmd, info_dict)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/external.py", line 372, in _call_process
    with TemporaryFile() as so, TemporaryFile() as se, \
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/packages/python/usr/lib/python3.8/tempfile.py", line 581, in TemporaryFile
    prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/packages/python/usr/lib/python3.8/tempfile.py", line 117, in _sanitize_params
    dir = gettempdir()
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/packages/python/usr/lib/python3.8/tempfile.py", line 286, in gettempdir
    tempdir = _get_default_tempdir()
  File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/packages/python/usr/lib/python3.8/tempfile.py", line 218, in _get_default_tempdir
    raise FileNotFoundError(_errno.ENOENT,
FileNotFoundError: [Errno 2] No usable temporary directory found in ['/data/data/com.termux/files/usr/tmp', '/']

[JunkFood02]

problem with No usable temporary directory found in ['/data/data/com.termux/files/usr/tmp', '/'] for now this problem can be solved by replacing /data/data/com.termux/files/usr/tmp with /storage/emulated/0/tmp in the tempfile.py in libpython.zip.so (May not work on Android 11+ version).

Please take a look here. It seems like we disabled the "cache"(or temp?) directory by default. Could you please try enabling this and pass the directory Environment.getExternalStorageDirectory() to it?

Edit: I see the problem is come with the temporary files. Manually specifying the -P directory may work

[pukkandan]

File "/data/user/0/com.yausername.youtubedl_android_example/no_backup/youtubedl-android/yt-dlp/yt-dlp/yt_dlp/downloader/external.py", line 372, in _call_process with TemporaryFile() as so, TemporaryFile() as se, \

The issue is coming from Python's tempfile module, nothing to do with --cache-dir. Does android have no temporary directory? Adding a dir to the TemporaryFile invocations should work around it

cc @lesmiscore

[JunkFood02]

The issue is coming from Python's tempfile module, nothing to do with --cache-dir. Does android have no temporary directory? Adding a dir to the TemporaryFile invocations should work around it

Yes, Android doesn't have a system scope temporary file directory. And from Android 11+ on, the application can only access the file/directory within the application data directory or some limited shared directories.

[Lesmiscore]

These TemporaryFile() are for capturing stdout/stderr without stucking Python interpreter, so it's not necessarily in temporary directory (although shouldn't be persisted) They can be converted to normal open() calls pointing to files inside Environment.getExternalStorageDirectory() and os.remove to do the same

[JunkFood02]

Aria2c and the progress text parsing looks well 👍

Screenshot