Tzahi12345 / YoutubeDL-Material

Self-hosted YouTube downloader built on Material Design
MIT License
2.6k stars 268 forks source link

Default settings (-f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4) only download 720p or lower anymore. #76

Closed GlassedSilver closed 4 years ago

GlassedSilver commented 4 years ago

I think by default youtuble-dl grabs best quality anyhow, so maybe setting this is kinda passé anyhow?

A setting to change this globally (subs and on-demand) would be helpful. I'm open to grabbing VP9 or AV1 as well if the codecs are provided.

I'm a bit busy atm, but there are even more intricate ways to figure out the best quality. It isn't always what youtuble-dl auto-picks, but personally I'm already pretty fine with whatever is the highest res.

Maybe an option to prefer open codecs could be nice for those who fancy the ultimate independence? :)

Ok, maybe a bit of a tangent there. Let's leave it at a bug report for now. :)

Log with two attempts:

2020-05-08T06:46:40.211Z INFO: YoutubeDL-Material v4.0 started on PORT 17442
2020-05-08T07:21:49.042Z ERROR: Unexpected end of JSON input
(node:1) UnhandledPromiseRejectionWarning: ReferenceError: req is not defined
at Readable.<anonymous> (/app/app.js:1425:38)
at Readable.emit (events.js:228:7)
at endReadableNT (_stream_readable.js:1185:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:1) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
(node:1) UnhandledPromiseRejectionWarning: ReferenceError: req is not defined
at Readable.<anonymous> (/app/app.js:1425:38)
at Readable.emit (events.js:228:7)
at endReadableNT (_stream_readable.js:1185:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
(node:1) UnhandledPromiseRejectionWarning: ReferenceError: req is not defined
at Readable.<anonymous> (/app/app.js:1425:38)
at Readable.emit (events.js:228:7)
at endReadableNT (_stream_readable.js:1185:12)
at processTicksAndRejections (internal/process/task_queues.js:81:21)
(node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 2)
GlassedSilver commented 4 years ago

Curiously I could pick 1080p for the video in the quality selector and it would work iirc, but the issue is that a) manual selection suuucks and b) it doesn't go higher than that. So those 2k, 4k or higher videos... Nada. No selection for that.

Tzahi12345 commented 4 years ago

Hm, a couple things are going on here, but I'm still not sure why it's not getting the best quality.

  1. I forgot (again) to update the simulated args so that may not be reflective of what's actually being run under the hood. I'll have to refactor the way this is done to avoid having to update it manually every time the logic changes in the backend. Basically, I'll make it so the simulated args aren't "hardcoded" per se, and instead asks the backend what the args would actually be. Seems like a relatively simple fix, I'll get this done ASAP to avoid any more issues on that front.

  2. Related to above, for most video downloads, the default quality changed from -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4 to -f best[ext=mp4]. This is where the problem might be coming from, perhaps the latter string gets the best format already as an mp4 rather than going for the higher quality format and then converting it to an mp4.

I'll do some testing, but if my guess is right, I'll opt for -f best instead and if the extension happens to be mkv or webm I'll do some conversion magic and get them to the right format.

So those 2k, 4k or higher videos... Nada. No selection for that.

This I'll also look into more. Right now it's set up to recognize 4K and 1440p as valid formats, so they should appear as options. Admittedly, I just tried this with an MKBHD video and didn't get any options. I looked more into it, and I think I know why this is happening.

For "safety" reasons only mp4 formats are available for selection. This works fine for most videos, but I think for videos with a higher resolutions, the mp4 resolution you want is probably not available. Here's the resulting quality options for that 4K MKBHD video:

Only 960p, 640p, 428p, etc. are available. If I add in the logic I discussed above for converting video files to mp4s, I'll be able to include all the formats and so the higher quality options should appear.

Maybe this is more long-term, but it might be a good idea to remove this dependence on the mp3 and mp4 file types. Webms are kind of unwieldy so maybe those can be handled differently, but in general it feels silly to be so limited on file type and extension.

I'll see what I can do this weekend on the not-so-hard stuff and get higher quality videos working again out of the box. But I like your idea of an easy way to modify the default quality select. I could have global args take care of it and just overwrite the current default format, but a dedicated setting would be more intuitive.

On the codec front, I could imagine a setting where you can rank codecs based on preference. It'll probably act as a soft preference, in that if you want the "Max" quality with your first preference as VP9, it won't try to get the best VP9 available. It'll instead look at the max resolution formats and go down your list of preferences until it gets the specific format that uses that codec.

GlassedSilver commented 4 years ago

Another thing I noticed, if I do select 1080p it will usually download in 1080p30fps when 60fps is available as well.

Hmm...

Tzahi12345 commented 4 years ago

Hm, that's something to check out as well. Right now, I don't check for fps when selecting a format, but that's something that should be done.

I may delay this issue until 4.2 as I faced several issues trying to allow for any resolution:

So my plan for resolving this and a couple other issues we're having is to switch the main library we use for video downloads. I made an issue for this which you can see here. Let me know if you have any thoughts on a stopgap solution until that gets done. Maybe the fps issue can be resolved independently.

powerdude commented 4 years ago

Sorry to interrupt this thread, but how do i change the resolution from best to 720p for subscription? Also, on a tangent, is it possible to specify a different filename? I'd like to rename the file to make it more friendly for Plex to something like this from another tool i was using:

--output '/workdir/%(uploader)s/%(uploader)s - S01E%(playlist_index)s - %(title)s.%(ext)s'

Tzahi12345 commented 4 years ago

@powerdude No problem, at some point GitHub discussions will come out and that will be a much better place to ask questions, but I digress..

but how do i change the resolution from best to 720p for subscription?

In the latest nightly (use Docker tag nightly) you can set custom args when you subscribe. Set this as your custom args -f,,(mp4)[height<=720] and it will limit the resolution to max 720p. Sometimes 720p isn't available and it will grab the resolution just below it.

Also, on a tangent, is it possible to specify a different filename?

Yup!

In the latest nightly, you can also set a custom output. To follow the format you wrote above, I would set %(uploader)s/%(uploader)s - S01E%(playlist_index)s - %(title)s as the custom output. Your videos for that playlist would be organized into separate folders for each uploader in the subscriptions/playlists/{playlist_title} directory, and the filenames would follow the format you described above.

Let me know if you need any more help!

shomilsaini commented 4 years ago

Good Effort guys,

Curious as to why only 720p is the max output even for 8K Videos. ( Tested with - https://www.youtube.com/watch?v=1La4QzGeaaQ ) Can we have a Quality option on the download page. And a title option similar to youtube-dlg. Then it be my permanent docker.

Cheers!

GlassedSilver commented 4 years ago

Good Effort guys,

Curious as to why only 720p is the max output even for 8K Videos. ( Tested with - https://www.youtube.com/watch?v=1La4QzGeaaQ )

It has been explained in this thread already, that this is a bug and will get fixed probably when the youtubedl library is changed.

Read the second post in this thread for some background info.

Can we have a Quality option on the download page.

There is one though... Or do you mean subscriptions' download page? There it's still lacking, indeed. One-off downloads allow for quality selection right next to the URL text field. (default is "Max", which as we have learnt is buggy and will usually be limited to 720p or 1080p 30fps. At least atm)

And a title option similar to youtube-dlg.

Titles are set according to what the original source provides. I have already suggested we add more flexibility in issue #138 .

If however you want completely custom titles then I think that's something that would still need to be added separately.

Keep in mind that things like categories and tags are somewhat on the horizon as well (planned, but no specific timeframe). So if you'd happen to use titles for structuring, you could achieve this with greater flexibility as well later on through those features. Setting completely custom titles however isn't a bad idea though, especially when you pull a video from a third-party source that doesn't provide a title or not a proper one. (e.g.: it's a Youtube re-upload and you know the original title and don't like the one the re-upper used)

That being said, I cannot speak on the behalf of Tzahi, just talking from user to user here. :)

Then it be my permanent docker.

Cheers!

Yeah getting real close to give it that treatment myself, too. One of my favorite pieces of software of 2020 so far. Especially the feature roadmap is looking bright and fills the gaps that all the other youtube-dl GUIs either ignore or conceptually miss entirely.

Cheers to you as well :)

powerdude commented 4 years ago

@Tzahi12345 thank you for the suggestions on renaming. I thought it was working fine, but just noticed a peculiarity. It seems to restart the playlist index every time it downloads a subscription, so not quite sure how it decides this. Another tool i was using was using the correct playlist index, but this seems to be doing something different.

Tzahi12345 commented 4 years ago

You're saying when a new video downloads for the subscription, the index starts at 0?

Is this subscription a channel or playlist? If it's a channel, this might be because the new videos get placed as playlist_index = 1, so it appears like it's resetting. Here's two ideas:

  1. Add --playlist-reverse as a custom arg for the subscription and each new video will (probably) get the next highest playlist_index, thus avoiding the reset issue (though your index will be in reverse order).
  2. Maybe instead of playlist_index you can use %(timestamp)s. The higher the timestamp, the more recent the video is (so again reverse order)

Not sure how the other tool handled this, if it's open-source, share the project so I can find out how they did it. And let me know if any of these fixes give you trouble!

GlassedSilver commented 4 years ago

Say we download videos out of order manually... is the index for the channel order going to be screwed as well for this instance? (I never really understood why other tools like the youtube metadata agent for Plex has such flaky index support... Just check which position the video is at in the stream of uplodads from that channel??? I must be missing something here. In the case of Plex downloading and adding videos from a channel out of order would screw with thumbnails.)

powerdude commented 4 years ago

@Tzahi12345 your suggestion works. It was what the other tool was doing behind the scenes.

im-n1 commented 4 years ago

Just ran into this and I confirm that. When I turn on Advanced settings and try to run the printed command manually I get desired output. Thats strange.

command:

youtube-dl https://www.youtube.com/watch?v=R94ntpWVelw -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4 -o video/%(title)s.mp4 
Tzahi12345 commented 4 years ago

@im-n1 Can you try enabling "Safe download override" in the Downloader tab in the settings and see if that helps get the desired output?

im-n1 commented 4 years ago

@Tzahi12345 Just tried - no difference.

I also noticed that once I paste video URL into to form field app tries to load (a loading circle appears) possible qualities to the dropdown but after like 4 secs it disappears and no quality options besides "Max" are available.

BTW: running youtubedl on ancient Python 2 might be the issue.

im-n1 commented 4 years ago

Can you please release ARM version of the latest version which contains this fix on Docker Hub?

Tzahi12345 commented 4 years ago

@im-n1 Just pushed an updated build! Still trying to fix autobuilds, once that is done both versions will be in line with each other.