search

abayochocoball / hollow_memories

Save your precious memories.
51 stars 16 forks source link

Automation best practices #8

Open abayochocoball opened 4 years ago

abayochocoball commented 4 years ago

Write about how to automate archiving process.

Starting point: Task scheduler/cron

GoldElysium commented 4 years ago

Automating it consistently is a horrible thing to do, YT will detect these patterns and block you from downloading. (Speaking from experience). Possibly immediately download when a stream starts could be a work around?

abayochocoball commented 4 years ago

When you were running a schedule, how often were you polling for updates?

GoldElysium commented 4 years ago

Weekly in my case

SusuTawar commented 4 years ago

immidiate download is also detectable pattern, though i'm not sure if youtube can tell the difference between watch and download.

i have a few things in mind using PubSubHubbub its server to server push notification, not sure how it behave on live schedule i can test it may took 24hours or so

GoldElysium commented 4 years ago

It seems YT can detect it, other people noticed that it blocked ytdl but they could watch videos fine

SusuTawar commented 4 years ago

update, i did an oppsie by using wrong parser. but the notification are coming.

so automation "may" be is possible. by utilizing server that recieve notification as middleware to broadcast youtube link to automated script that call youtube-dl in the client computer.

i can write the server files and maybe the client script my only concern is youtube ToS, if its not allowed then i wont continue.

abayochocoball commented 4 years ago

https://www.youtube.com/static?template=terms

access the Service using any automated means (such as robots, botnets or scrapers) except (a) in the case of public search engines, in accordance with YouTube’s robots.txt file; or (b) with YouTube’s prior written permission;

That isn't going to stop me from writing about it though.

GoldElysium commented 4 years ago

If you're interested in the PubHubSubBub thing, I've got some code. It's for a different purpose(notifs in a Discord server), but can be used. It's still in a PR and needs to be tested, but here is the branch: link. If that link doesn't work, here is the master branch: link. Let me know if you have any questions.

SusuTawar commented 4 years ago

the pubsubhubbub (let's call it websub) is working perfectly, i tested it for about 12-17 hours. i get all kind of notification privated video, stream schedule, stream start, stream/video name change. which can be filtered later using youtube-dl

also its totally possible to create a server to forward the notification to python script (where youtube-dl reside) to automaticly start the download.

maybe we can get around the "automated means" by asking for user action before start downloading ?

SusuTawar commented 4 years ago

server are done, havent tested it. i'll try work on the download script, dont put much hope it wont have any gui. client script

on that note can anyone help to fill the data in channels directory ? still missing out a lot.

GoldElysium commented 4 years ago

@SusuTawar You get stream start notifications? You sure? I'm pretty sure you don't, at least I don't get any and I can't find anything about it either. I've been writing code to work around it. Also, I've tried using your HMAC verification code but it fails for me.

abayochocoball commented 4 years ago

I'll start looking into this soon. Currently trying to create a very simple timer based solution and detect for collabs based on video description.

SusuTawar commented 4 years ago

@GoldElysium i dont. but client script check if its live/downloadable or not. the error handling pretty straightforward with youtube-dl. the only thing that i havent sure yet is video name change and video upload. as it is now it'll find live metadata from youtube-dl before start downloading, so no haachama cooking.

HMAC ? maybe the secret ? i didnt include the secret in the code. but i make sure its working before working on anything else, havent touch it since then. EDIT: yeah i made sure secret is required when subscribing, just in case.

i cant really continue for maybe a week, got my laptop broke and my phone dead..

GoldElysium commented 4 years ago

i dont. but client script check if its live/downloadable or not. the error handling pretty straightforward with youtube-dl. the only thing that i havent sure yet is video name change and video upload. as it is now it'll find live metadata from youtube-dl before start downloading, so no haachama cooking.

So you can't handle planned livestreams?

HMAC ? maybe the secret ? i didnt include the secret in the code. but i make sure its working before working on anything else, havent touch it since then.

Weird, I'm sure I include the secret but it doesn't work in my code, will look into it a bit more.

i cant really continue for maybe a week, got my laptop broke and my phone dead..

F

SusuTawar commented 4 years ago

i just get my phone back, i planned livestream work its usually has 3 or 4 notification depend on the channel but only 1 that has is_live metadata. and that is when the download start