search

Tzahi12345 / YoutubeDL-Material

Self-hosted YouTube downloader built on Material Design
MIT License
2.67k stars 275 forks source link

Large volume of requests for subscriptions #475

Open jm2003uk opened 3 years ago

jm2003uk commented 3 years ago

When viewing queries on pi-hole, there seems to be a large number of requests when subscriptions are being checked (originally mentioned in #467). These were the results of a 24 hour period:

Screenshot 2021-09-29 19 06 25 Screenshot 2021-09-29 19 07 26

I've done a few tests this morning by starting a new instance and flushing the pi-hole logs.

Screenshot 2021-09-30 07 48 51

This was with one subscription added which itself only has 6 videos and set to download videos uploaded within last 1 day, of which there were none.

Screenshot 2021-09-30 07 46 36

This was with one subscriptions added which has 66 videos. Download options same as above, within last 1 day, of which there was also none.

Screenshot 2021-09-30 07 55 21

This final one was no subscriptions and downloading a video directly via url.

Could it be that when a subscription is added and it starts checking, that it goes to each video page individually? It would make sense to be able to get the description and other info and would certainly mean that the more videos there are, the more requests there will be. But if a video should only be downloaded if it's less then 'x' days old, once it gets to one older should it not just stop all further checks? Say, it's set to download videos less than 8 days old, the first video it comes to that's 9 days old, it stops and ignores everything else.

dejan995 commented 3 years ago

From my point of view the numbers from the black pictures look normal. Not sure though. The explanation and math that @Tzahi12345 provided in the previous thread were new to me too. I did not know that the check interval option was behaving like that.

Going with that math maybe you can set the option to be 172800 seconds. This will check one subsription every hour. The bad part is that it will need 48 hours to check all off your 48 subscriptions you mentioned previosly. This should even out the requests being sent out and you should see an occasional spike every hour, though it will be much much smaller.

@jm2003uk A small side note. Pi-Hole is a network-level DNS AdBlocker. Are you sure that all of those requests are comming from the YoutubeDL-Material app and not from something else on the network. I run Pi-Hole myself too and I'm nowhere near seeing those numbers of requests and also my family are heavy YouTube useres.

jm2003uk commented 3 years ago

@dejan995 Fairly certain. All of those images were on a network made up of my phone, chromebook and the pi. It might be that it has no major effect on anything, it just seems strange that there are so many requests when, in my case, it was only looking for new videos within the last day. Best way i can think to try this is pull up either pihole logs or the main dashboard, get a feel for what the normal traffic is like and then add a subscription with decent amount of videos.