[FEATURE] 429 Rate limit handling

[jonathanm-ur]

Is your feature request related to a problem? Please describe. When youtube receives too many requests in a period of time, an error is returned instead of the desired content.

Describe the solution you'd like I'd like

1. When a 429 is returned, pause all pending requests and start an incremental timer, reset when 429 is no longer returned. So, when the first 429 is detected, pause for 60 seconds, try again. If another 429, increment the counter, and multiply 60 by the the counter value. So if 5 consecutive 429's are returned, the next request will happen in 300 seconds. The initial delay should be user configurable, so they can set something other than 60 if desired.
2. When the delay is in effect, display a persistent GUI warning across the top of every page, with a message that "Your WAN IP (123.456.789.123) has been temporarily blocked by Youtube. Next retry in XXX seconds, or press here to retry now". A button press would trigger a redetection of the WAN IP and a reset of the counter.
3. Log specific error every time 429 is caught, with time to retry and current WAN IP

Additional context Currently, I run a script that checks the last line of the log files for HTTP 429 every 5 minutes, and if it's detected, it shuts down this container, restarts the VPN container that provides the network, and starts this container back up. That generally works, but it would be nice to have a way inside the container to rate limit instead of just hammering away and totally killing your current WAN IP.

[GlassedSilver]

I like this idea, however those pauses should only be scoped for YT, because YT slowing you down does not affect your Twitch and other supported sites' queries.

Edit: the new Download Manager #420 that's a work in progress might make this unnecessary or a much rarer case, since the amount of concurrent downloads will be managed by it, greatly increasing resilience against rate limiting.

[jonathanm-ur]

I like this idea, however those pauses should only be scoped for YT, because YT slowing you down does not affect your Twitch and other supported sites' queries.

That's why every reference in the feature request is Youtube. I guess I should have put that in the title of the request.

Edit: the new Download Manager #420 that's a work in progress might make this unnecessary or a much rarer case, since the amount of concurrent downloads will be managed by it, greatly increasing resilience against rate limiting.

Hopefully. I'm not holding my breath though, since it's so hard to pin down when you are going to get 429'd. I've seen it happen without any downloads queued, just as a result of querying the channels for updates. It's possible the extra activity generated by scraping the channels for content without actually starting the downloads will trigger it.

[GlassedSilver]

I've noticed that myself, that a lot of channel subs are more likely to yield it, but oddly enough using YT in the browser with the same WAN IP at the same time was absolutely possible.

Sidenote: Can we please appreciate that the issue number for this ticket is 429 much like the error code you report? I find that mildly satisfying

[jonathanm-ur]

Sidenote: Can we please appreciate that the issue number for this ticket is 429 much like the error code you report? I find that mildly satisfying

Why do you think I submitted this one first instead of the dup feature? I've been waiting for this opportunity. ;-)

[jonathanm-ur]

oddly enough using YT in the browser with the same WAN IP at the same time was absolutely possible.

I've gotten redirects on a web browser to a captcha when this occurred.

[GlassedSilver]

oddly enough using YT in the browser with the same WAN IP at the same time was absolutely possible.

I've gotten redirects on a web browser to a captcha when this occurred.

Maybe YouTube just loves me or something, that being said, I also noticed that YT is quick to let loose again a little bit, probably because they absolutely expect some IPs to hammer away quite aggressively. Cases where student dorms and places like these all share on WAN IP and generate dozens or maybe even hundreds of calls to YT at the same time.

You really gotta give it to YT, they are pretty realistic about their own site's importance and impact when weighing measures like these, but playing around with the interval value as well as waiting for the new download manager component might prove fruitful. In any case I think it's definitely good if @Tzahi12345 chimes in as well on this one. :)

[Tzahi12345]

Hey, sorry for the delay in responding, I think this is a great idea and actually a duplicate (or rather more fleshed out suggestion) of #173. Of course I can't close this because, well, it's issue #429 but I do like your suggestion that you put there. I think we need a notifications system to display something to the user, but I think we can at least have a button at the top for now, or maybe a popup at the bottom (called a snackbar) that shows up on page load.

So I think this ultimately has to be integrated into the maintenance dashboard (#422), it can be a "task" that doesn't run recurringly, rather gets generated as "finished" with the action start downloads again. In the meantime, all downloads and subscriptions will be paused. How does that sound?

[GlassedSilver]

Silently pausing my downloads? If we limit the scope to just YouTube, sure.

One thing I'm wondering about though, you'd have a cool down for it as well? So even in a headless use case (idk, gone for holidays just letting your server do its thing) it'd still re-enter downloading mode.

Maybe make auto-pause a toggable option. Not sure what the default should be. With good scoping and with cooldown to auto-disable the pause a default of yes sounds good.