[Feature / ?] Migrate from yt_dlp to InnerTube & Update structure of the project

[ii7mood]

This would be a massive change allowing for all sorts of features and implementing something like this would require a decent chunk of the code to be re-written (Detector.py & streamers.py & streamers_vars.py) but I have no idea how to reverse-engineer private APIs such as this. I will look into it over the coming... days? weeks? months? Who knows?

I will look into it but I wouldn't get my hopes up. There is a library that allows us to interact with the InnerTube API but there are no docs (understandably, these APIs change their structures frequently). This might be a new learning experience :)

There is a Javascript implementation of a wrapper of the InnerTube API which is quite interesting and it comes with docs but no one can pay me enough to use Javascript, I am too much of an idiot for it (confusing af). Anyway, I'll look into it™

[ii7mood]

uhh so I got something working. I am able to extract metadata of videos through InnerTube. There's still a lot to go over but I think I can (possibly) do this?!

Literally just read these blog posts by Tyrrrz to get an idea of how this works post #1 post #2 and then refer to yt-dlp for the rest.

I haven't really learned much though. There's still a lot I need to understand so I think for those I will try to tackle them myself.

[ii7mood]

Okay. This is huge. I can essentially retrieve all streams' metadata including their video IDs through a single request. Sanitising the data is what's going to take a bit of work but that's not a huge problem. This will allow us to do all sorts of stuff.. hell yeah

[ii7mood]

I will be doing a somewhat big re-write of the backend to use InnerTube API. We will be removing yt-dlp completely as a dependency meaning we'll have to force using Twitch API.

I will be re-writing the Twitch backend as well to make use of implicit grant flow meaning the user won't really need to create an application or anything, just authorise our app.

This will take a while so you might not see any commits for ~2 weeks.

[ii7mood]

The /browse endpoint expects a channel ID. I am not quite sure how to get that from the channel handle. I tried looking for something like a /resolve endpoint but that doesn't exist. For now I'll use the /search endpoint giving the handle as a query. Usually the first result is the channel and includes the channel ID which we can use as the browseId parameter. yt-dlp's code is incomprehensible to my feeble mind.

[ii7mood]

Seeing as how different versions of InnerTube clients yield different responses from the API I think it would be better for me to just use a single web client. If necessary we'll probably have to do something like yt-dlp's traverse_object() but I don't see why we would need more than a single client... ¯\_(ツ)_/¯

Honestly if there's something I have learned it would be how huge yt-dlp actually is. I learned so much from just reading it's code, even if I don't understand it ¿?

[ii7mood]

I will be creating an MR in the coming days as this a massive change and requires lots of testing so I would rather not push it straight into the main branch. The TwitchC changes will be going straight to main though.

Instead of yt-dlp we will be using YouTubeC which is a """wrapper""" I wrote for InnerTube and has definitions for things like _getStreams(), _getChannelMetadata(), _getBrowseID() etc.

Most functions will need the UCID of a channel (User & Channel ID) to work hence the need for _getBrowseID(). UHhhhh maybe another week I guess? Haven't had a lot of time to work on this soz.

[ii7mood]

I am not quite sure how safe it is to send automated requests using data from cookie files. I will limit usage of cookies as much as possible in hopes that it does not trip any automated ban or something. That means the addition of a subscribed field to the DB. We will be sending a single request using data from the cookie to every streamer who's subscribed field is True every five minutes.

At minimum every streamer will require a single API request, and at maximum two or three requests (of which only one will make use of cookies) every iteration. 90% of the time it will be a single request per streamer so... I think it should be safe.

I'll make #6 (detect member-only streams) low-priority right now as there are still several major things that need to be done, incl re-structuring the DB.

[ii7mood]

It's been two weeks, been busy. Pretty much 80% of stuff is done. Gotta clean up the code and implement the little stuff and I will then create an MR and after testing it I will merge. This is pretty much an entire re-write of everything except the Listeners. My schedule is coming week is unpredictable so I don't really have an estimate.

[ii7mood]

pushed commits to a testing branch. Also the project is slightly changing it's structure. LiveMood is now simply a server of metadata and Listeners (such as the Discord or Desktop Listeners) will move to their own separate repositories. Will start testing for a week or so and update the testing branch till its stable wherein i'll merge it into the main branch.