search

Tzahi12345 / YoutubeDL-Material

Self-hosted YouTube downloader built on Material Design
MIT License
2.62k stars 271 forks source link

[BUG] Not Downloading Full Playlist #458

Closed rockbo842 closed 3 years ago

rockbo842 commented 3 years ago

Describe the bug I've noticed that YDLM isn't downloading the full content of my playlists. For example, I have one playlist that is 62 videos long but it only downloaded 20 of the videos. I thought maybe it was not going back to download videos that may have been blocked when YouTube decides to shut me down. However, it does not add new videos when I've put them into the playlist during periods that I know YouTube has unblocked me.

Expected behavior YDLM should pull all videos from the playlist.

Screenshots If applicable, add screenshots to help explain your problem.

Environment

Additional context Is there a setting that I may have screwed up that is causing the issue? Thanks.

GlassedSilver commented 3 years ago

One thing I noticed is that a few things don't download for me as well.

I have the sneaking suspicion that it's me still using youtube-dl as the downloading agent. Try changing that one and see if that helps.

You can always attach logs as well if you want, that might help circle it in as well. :) (Settings > Logs)

Bonus tip: when using yt-dlp I'd suggest to use --no-clean-infojson as custom argument. Feel free to add it or not, but it might be required for some future features of YoutubeDL-Material. (relevant docs if you want to reference) Not sure if this is one of the default arguments we pass already when using yt-dlp. @Tzahi12345 ?

If not, we should be using it. :)

rockbo842 commented 3 years ago

I didn't know there was another agent=) How do I change it? Have you found one that works better? Logs.txt ?

GlassedSilver commented 3 years ago

I didn't know there was another agent=) How do I change it? Have you found one that works better? Logs.txt ?

Settings > Advanced > first setting called "Select a downloader" :)

I think many already switched to yt-dlp. As I said, I'm still on youtube-dl myself, but mostly because I'm very conservative about switching forks. I'm still figuring out my desired settings from yt-dlp's docs before I set YoutubeDL-Material to that myself.

Hope this helps. :)

PS: preferably you'd just copy and paste like the last 50 lines of logs into here and then wrap the selected text with the "code tag" so it looks

tidy and proper
like this and doesn't accidentally parse any special characters as formatted text. :)

Click this button when selecting the quoted log part: image

That way any possible hints are fully searchable as well for future reference and others maybe stepping into similar problems. :)

GlassedSilver commented 3 years ago

Anyways. I looked at the log and I do get these myself.

We're aware of this problem and as I said, I will soon evaluate whether yt-dlp is the solution, feel free to try before I do though, because others seem pretty happy with yt-dlp, as I said however, setting that one custom arg would be advisable.

Tzahi12345 commented 3 years ago

I'll take a look at this error tonight, if it doesn't exist on yt-dlp I'm going to make that the default to avoid any errors.

Not sure if this is one of the default arguments we pass already when using yt-dlp. @Tzahi12345 ?

We do already pass that in by default, so no need to include it :)

Tzahi12345 commented 3 years ago

Would you mind providing the playlist URL?

rockbo842 commented 3 years ago

Changing to yt-dlp did help significantly! I'm going to unsubscribe/re-subscribe to the playlists to see if that resolves the issue.

Would you mind providing the playlist URL?

I prefer to keep that to myself=)

Tzahi12345 commented 3 years ago

That's fair! Let me know how yt-dlp goes, if it runs without issues I'll change the default downloader.

rockbo842 commented 3 years ago

This downloader is much, much better. It's not perfect but it is grabbing many more videos from the subscriptions and playlists than the default downloader.

I am still getting some warnings:

WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. 2021-09-23T10:12:54.454Z ERROR: WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. ERROR: Postprocessing: Conversion failed! 2021-09-23T10:32:14.163Z ERROR: WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. ERROR: Postprocessing: Conversion failed! 2021-09-23T10:32:41.088Z ERROR: WARNING: [youtube] Unable to download webpage: HTTP Error 429: Too Many Requests WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. WARNING: [youtube] Cannot extract signature timestamp without player_url. ERROR: Postprocessing: Conversion failed!

GlassedSilver commented 3 years ago

My assumption is that since you were missing out on a lot of downloads recently that firing it up again does generate a lot of requests at once that got backlogged (naturally).

That amount is too high and you get rate-limited.

Give it some time and let it sit there. Eventually you'll be caught up again and stuff should (hopefully) flow better again. :)

rockbo842 commented 3 years ago

My assumption is that since you were missing out on a lot of downloads recently that firing it up again does generate a lot of requests at once that got backlogged (naturally).

That amount is too high and you get rate-limited.

Give it some time and let it sit there. Eventually you'll be caught up again and stuff should (hopefully) flow better again. :)

I think you're right. It has done that for most of the playlists and subscriptions. However, there was one playlist I was impatient on. I deleted it from the system and resubscribed thinking that the whole playlist would start over. Instead, nothing has been downloaded at all. I tried renaming it and still nothing. Do you have any thoughts?

GlassedSilver commented 3 years ago

Did you actually delete the subscription from the subscription page only or did you also delete any downloaded files?

Additionally, the video IDs might still be loaded as already downloaded.

rockbo842 commented 3 years ago

Interestingly, deleting the subscription from the subscription page seems to have deleted the folder (and videos) from my computer.

GlassedSilver commented 3 years ago

Interestingly, deleting the subscription from the subscription page seems to have deleted the folder (and videos) from my computer.

Expected behavior I think.

So did you just do that now then and can initiate a re-subscription successfully now as well?

rockbo842 commented 3 years ago

I did it yesterday and resubscribed but nothing has happened since. I just noticed the deleted folder and videos a few hours ago.

GlassedSilver commented 3 years ago

Think the archive file for it didn't get cleared.

Can you check if the following folder contains an archive.txt file with a load of video IDs in it?

/subscriptions/archives/<NAME OF YOUR PREVIOUS SUB YOU DESIRE TO BE REMOVED>

rockbo842 commented 3 years ago

This is a silly question but I'm still learning. Would the /subscriptions/ folder be on the cache drive under appdata on Unraid?

GlassedSilver commented 3 years ago

This is a silly question but I'm still learning. Would the /subscriptions/ folder be on the cache drive under appdata on Unraid?

There are no silly questions. The whole point of a question is to gather information you're missing, no shame in that!

Me rambling about a bad experience having my download share cache-enabled (Well I hope you're not downloading to your cache drive, unless your downloads share is cached itself, but targets (gets moved) to the array. Even that... wouldn't recommend. I've learned the hard way when my YoutubeDL-Material had to catch up with one month's worth of downloads after some downtime. Filled my cache drive real quick and weird stuff happened. :D)

Either way, here's how you find out where your subscriptions get stored on your host:

  1. Go to your Docker page in unRAID, then find your YoutubeDL-Material container in the list.
  2. Click on its icon
  3. select "Edit" image
  4. Refer to what's listed here: image

Hope that helps. :)

rockbo842 commented 3 years ago

Yeah, I started seeing weird stuff going on so I just made a separate share on the array without the use of the cache and changed the download destination to it.

My folder for Subscription Downloads is the same as my Video Downloads folder. However, I don't see an archive.txt there. I checked in Windows with hidden files on and in Dolphin.

GlassedSilver commented 3 years ago

Ah could be you're not using the archive feature. Can you verify there's no tick with this setting on your end?

In YoutubeDL-Material's WebUI:

  1. Settings > Downloader
  2. scroll down image
  3. Is "Use youtube-dl archive" ticked?
rockbo842 commented 3 years ago

That was unticked. I have not ticked it.

GlassedSilver commented 3 years ago

Hmm, because that would usually be pulled for a check whether we downloaded something before.

Maybe the database has those entries still.

You can inspect it with MongoDBCompass, MongoDB's official "workbench" tool.

If you need help connecting to your db I can provide that and then we figure that out together what could be off, however I think @Tzahi12345 might have more of a direct clue without much poking around since he is actually coding all the time, I'm just a super dedicated user and do some support :P

rockbo842 commented 3 years ago

Thanks for the offer and all of your help!

I decided to remake the playlist in Youtube so I could have a fresh URL and see what happens. If it continues to be a problem, I will reach out and take you up on the db help.

On a separate note, if I want to learn to code and make things like this app, where do I start?

rockbo842 commented 3 years ago

Update - Creating a completely new playlist seems to have done the trick.

rockbo842 commented 3 years ago

Thanks, @GlassedSilver and @Tzahi12345 for all of your help. I'm still curious about coding advice if you two have any to give.

Tzahi12345 commented 3 years ago

If you need help connecting to your db I can provide that and then we figure that out together what could be off, however I think @Tzahi12345 might have more of a direct clue without much poking around since he is actually coding all the time, I'm just a super dedicated user and do some support :P

Not true, you're a code contributor now! :)

@rockbo842

Glad it worked!

There's no easy answer on how to pick up coding and create a project like this. I think a good goal is to just learn programming through a project you feel passionate about. It's not about the end goal but the journey, right?

I'd recommend Python as the syntax is easy and you can quickly jump into core concepts like variables, functions, and classes. Here's a course I saw recommended on Reddit.

Anything beyond that depends on what kind of project you want to do, but if you want to do a youtube downloader kind of thing, here's an idea:

A python script that asks you to input a youtube link, and then downloads the video. Here's some example code that might be helpful.

Check out those videos, if you run into any problems feel free to reach out to me on Discord (Tzahi#8672) and I can provide some tips. Programming is fun but it can get frustrating, especially at the start so don't hesitate to ask for help!

GlassedSilver commented 3 years ago

I'm on my break at work, but if I had to shoot an idea right now I would suggest you start with bash scripting.

My first contact with coding was LEGO Robotics at high-school and that's also when I had a semester's worth of a university beginner's C++ course.

Nothing ever got me quite as hooked as scripting for the quick and immediate success.

The first time I legitimately had an urgency and a passion to come up with a fix to a problem was when I needed to unflatten a folder with a bash script to help my "OCD"...

Fast forward to today and I'm working on a management suite for a server software I run (danbooru) locally, automate and aspire to write checks for other self-hosted apps and services and... yeah I'm slowly and surely really getting super amped about bash scripting.

Later on I might add more Powershell, already dabbled in it a bit and made a little helper to partially automate some ingestion steps for when I want to feed that danbooru server some content :p

But you know, in the end it's best to start off with something semi-simple that you might find some code snippets for that you just try to dissect and understand. Then you slowly expand functionality and make it bigger and more beautiful.

As for a language... After getting more into scripting in bash and PS I'll consider Python myself as well as Javascript.

In any case it should always be fun, take breaks when stuff seems overwhelming, but do document your code well!

Nobody likes going back to some weeks old code and ponder what the hecc they were doing there and having to refigure it out. yuck!

Oh and modularize your code! You'll love and thank yourself later for it manyfold! :D

Glad you fixed the issue by the way. It's nice when stuff works. :p

rockbo842 commented 3 years ago

Thanks to both of you. I had heard that Python was easier to learn than some other languages. I'll start with bash scripts and Python.

cyberskycat commented 2 years ago

That's fair! Let me know how yt-dlp goes, if it runs without issues I'll change the default downloader.

yt-dlp work better than the youtube-dl and youtube-dlc