youtube-dl backend

[Atemu]

Due to the nature of its task, NPE often has major issues which is very detrimental to the UX.

What would you think of an (optional?) youtube-dl-based backend? Their extractor is usually more reliable and updated faster to accommodate for youtube's frequent changes.

The NPE would still need to be used for things that enable browsing the website which youtube-dl doesn't support and probably won't support in the future either.

It would also open up limited access to tonnes of other websites (viewing only, browsing isn't supported by youtube-dl).

Is this something worth pursuing?

The largest question mark is the integration of youtube-dl itself since Python and Android don't mix well but approaches have been demonstrated that wrap around a statically compiled Linux binary (https://github.com/mpv-android/mpv-android/pull/58) which is hackish but should work just fine.

Obviously there are size concerns, so I did a quick build with NixOS/Nixpkgs@d8f8f31af9d77a48220e4e8a301d1e79774cb7d2:

pkgsCross.aarch64-multiplatform.pkgsStatic.youtube-dl.override {
  ffmpegSupport = false;
  rtmpSupport = false;
  hlsEncryptedSupport = false;
}

$ nix path-info -S ./result/
/nix/store/dgpa0aam0dycfsaky86qaf32j8vdf6wn-python3.8-youtube-dl-2021.06.06-aarch64-unknown-linux-musl    161163952

So a closure size of ~153MiB without much optimisation. Obviously quite large for an Android app but not outlandish either; the official youtube app is >130MiB and the only optimisation done was to disable optional features in youtube-dl itself. Phones also have dozens of gigabytes of storage nowadays, so even a few hundred megs don't really hurt much.

Because of its size, youtube-dl should obviously have to live outside the NP package somehow. Downloading it at runtime would be an easy way but that sounds unclean, potentially dangerous and has a lot of overhead.
I'm not very well versed in Android IPC and the like but perhaps it could be spun into a separate app package? That way the URL youtube-dl supports could also be delegated to that package.
(Maybe the whole extractor could be like that anyways so that it's pluggable and the youtube-dl-based extractor would be a totally separate one?)

I'd really like to know what you all think of this.

[triallax]

I am not sure how much this would alleviate the size concern, but patching youtube-dl to remove all services/functionality not used in NewPipe should be at least somewhat helpful in that regard.

[triallax]

However, I definitely think that this SHOULD be optional. You say that a size of ~153 MB is not outlandish, but I strongly disagree with that statement.

[Atemu]

patching youtube-dl to remove all services/functionality not used in NewPipe should be at least somewhat helpful in that regard.

I'd look into stripping python of the parts that youtube-dl doesn't use first since we might want those extractors to be available and python alone is already 140MiB IIRC.

The youtube-dl + python used by the MPV PR is only about 13MiB unpacked, what I presented was an upper bound.

I definitely think that this SHOULD be optional.

Yeah absolutely. There are a ton of people who don't have access to phones with large amounts of memory. The reverse is also true though and I think these people would like to take advantage of such a feature.

[triallax]

Why would we want ALL the extractors to be available? I believe there are thousands of them (if not, well, a lot), and I doubt that we're going to be using all of them.

[Atemu]

Well, to play videos from all of those thousands of websites of course ;)

I don't think they're that large either and users who'd want to use this in the first place probably don't care about the few hundred KiB to a few MiB they'd take up.

[triallax]

If they indeed don't take up that much size, then yes, that makes sense.

[moriel5]

@Atemu I personally have noticed much better reliability with NewPipe Extractor, since YouTube-DL regularly drops connections for me, while NewPipe Extractor almost never disconnects in the middle of an operation for me.

[opusforlife2]

YouTube-DL regularly drops connections for me, while NewPipe Extractor almost never disconnects in the middle of an operation for me.

@moriel5 youtube-dl is only used to fetch the URL of the stream being played. Once it is fetched into the player, the youtube-dl binary's work is done and it closes. Then the streaming is done by the player itself (such as mpv-android). If your connection is dropping, it could be your player app, OS, or the network itself acting up.

[moriel5]

@opusforlife2 Thanks, so in that case, I should look into how NewPipeExtractor-based software manage the actual downloads, to see what you behind it's stability.

The strange thing, however, is that wget and aria2 are actually working fine for me, however downloads from YouTube, specifically when the recognition is made by YouTube-DL, keep dropping in the middle.