[Triage] Throttling?

[rjbutler]

Describe the bug I started using Pinchflat a few days ago. Since then I frequently get "Video Unavailable" for all videos I try when accessing YouTube from a browser and see the same error in the logs

If I stop the docker container, after some time I'm able to access videos again. I'm wondering if Pinchflat is being too aggressive. I have 36 sources, most of them are set for daily indexing. I have a cookies.txt file. I've tried this with and without an API key. I have a YouTube Pro account.

I used ytld-sub prior to this and never experienced the same issue. With ytdl-sub I had my subscriptions broken into 3 groups and ran 3 cron jobs daily to download them.

I prefer the web interface of Pinchflat if I can get it to work without experiencing this issue

19:00:02.263 [info] {"args":{"id":66304,"quality_upgrade?":true},"id":1630,"meta":{},"state":"success","max_attempts":20,"queue":"media_fetching","worker":"Pinchflat.Downloadin
g.MediaDownloadWorker","source":"oban","duration":1682308,"event":"job:stop","queue_time":579819,"attempt":1,"tags":["media_item","media_fetching","show_in_dashboard"]}

19:00:02.392 [error] [command_wrapper]: /usr/local/bin/yt-dlp called with: https://www.youtube.com/watch?v=YgbrG_72a3A --no-simulate --no-progress --force-overwrites --parse-me
tadata %(upload_date>%Y-%m-%d)s:(?P<meta_date>.+) --write-auto-subs --write-subs --convert-subs srt --sub-langs en --write-thumbnail --convert-thumbnail jpg --output thumbnail:
/downloads/Great Scott!/Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S99 %(title)S [%(id)S]-thumb.%(ext)S --write-info-json --clean-info-json --remux-video 
mp4 --format-sort res:1080,+codec:avc:m4a --sponsorblock-remove sponsor --output /downloads/Great Scott!/Season %(upload_date>%Y)S/s%(upload_date>%Y)Se%(upload_date>%m%d)S99 %(
title)S [%(id)S].%(ext)S --config-locations /config/extras/yt-dlp-configs/base-config.txt --print-to-file after_move:%()j /tmp/pinchflat/data/b32f800469be93fd056932ae6df800ac29
d76ecd58c5fd02ed9f22406686a69a.json --cookies /config/extras/cookies.txt --windows-filenames --quiet --cache-dir /tmp/pinchflat/data/yt-dlp-cache exited: 1 with: ERROR: [youtub
e] YgbrG_72a3A: Video unavailable. This content isn't available, try again later.

19:00:02.393 [error] yt-dlp download error for media item #52883: "ERROR: [youtube] YgbrG_72a3A: Video unavailable. This content isn't available, try again later.\n"

19:00:02.393 [error] yt-dlp download will not be retried: "ERROR: [youtube] YgbrG_72a3A: Video unavailable. This content isn't available, try again later.\n"

19:00:02.393 [info] {"args":{"id":52883,"quality_upgrade?":true},"id":1629,"meta":{},"state":"success","max_attempts":20,"queue":"media_fetching","worker":"Pinchflat.Downloadin
g.MediaDownloadWorker","source":"oban","duration":1820736,"event":"job:stop","queue_time":571817,"attempt":1,"tags":["media_item","media_fetching","show_in_dashboard"]}

Diagnostic info

• App Version: 2024.8.6
• yt-dlp Version: 2024.08.06
• Apprise Version: v1.8.1
• System Architecture: x86_64-pc-linux-gnu
• Timezone: America/Los_Angeles

[kieraneglin]

Hey there, thanks for the report!

That's definitely frustrating. I've set up the app to minimize concurrent yt-dlp requests to prevent this, but I actually don't think that's the issue. Myself and others have downloaded many thousands of videos without stopping and we don't see this error message.

What I suspect is the real culprit here is Youtube cracking down on yt-dlp. This has been an ongoing battle and in the meantime yt-dlp recommends against cookie auth in all cases (see "caution" blurb here). This applies to all downloaders that use yt-dlp (which is essentially all of them).

My recommendation for now is to disable cookie auth and see if that helps. If you need it, it might be best to move back to ytdl-sub for now while I look for ways to mitigate this issue

---

One solution I'm looking into an alternate method of authentication that some people say is more robust, but that's still in the research stage.

[rjbutler]

The only thing I need cookies for is downloading my "Watch Later" playlist. I could just use ytdl-sub for that, or even a second instance of Pinchflat I guess. Unless you think it would make sense to add a checkbox to the source indicating whether to use the cookies file?

[rjbutler]

One interesting thing is that this appears to be affecting the account, not my IP. I can switch accounts and YouTube works fine.

[rjbutler]

I think I will move back to ytdl-sub for now, but I’ll keep an eye on this project. Just a guess, but I wonder if the way you do indexing is contributing. I don’t think ytdl-sub does that, it just works backwards through the videos each time until it hits one it has already downloaded. I saw the issue during periods where only a handful of videos were downloaded, but indexing was ongoing.

I do like being able to look at the list of videos in Pinchflat though, so hopefully you can make this work.

[kieraneglin]

Fair enough! I'll keep this issue open and let you know once I've had a chance to take a look at it

[kieraneglin]

Actually, I have a little time today so I'll look into this. I'll keep you posted!

[kieraneglin]

I've made some improvements here in #354 and #355. I want to get a few other changes done first but that'll be going out with the next release!

[kieraneglin]

The new release with those cookie-related improvements is out if you want to give it a go 🤙🏻

[rjbutler]

I changed worker concurrency to 1 but left cookies enabled and it still happened. Removing cookies.txt seemed to prevent it.

The other thing I use cookies.txt for is for mark-watched. But I can probably do without that.

I'll experiment some more with enabling cookies for playlists.