subscriptions

[Randalix]

Hi Thank you for making ytfzf. Really enjoying it! Subscriptions would make it even more powerful. Could be a simple text file with channels in it and a command to view the latest videos.

Cheers Johannes

[cybardev]

Searching a channel name, for example "pewdiepie" already brings up the latest video of said channel under cursor, no?

Subscriptions can be added of course but if the above behaviour suffices, no need to further complicate things, no?

[Randalix]

Yes, this is great already. But what I mean is checking multiple channels at once.

[pystardust]

This is a requested feature There are two ways to implement this

1. get the channel rss
2. scrape the channel latest page

I feel the second way would be better, as it is easier to get the icons from there. I will add this to the todo list.

[cybardev]

checking multiple channels at once

Ok, that makes sense. So maybe something like this:

• When ytfzf is invoked with no flags (just by running ytfzf in terminal), or with some special flag (more flags... so many flags already... we'll have the whole alphabet at this rate), look for a plaintext (or JSON file), ~/.config/ytfzf/subscriptions.txt. Parse the file and get the top n (number to be decided) latest videos from listed channels and display them in the menu.
• About the subscriptions list file: what should it contain exactly (just names, channel ID, link, etc.) and how should it be generated? Should the user create it in a text editor, or import the JSON subscriptions from YouTube (would probably contain more detailed and well-formatted data and thus be easier to implement in the script, but it'll probably be bloated since people often subscribe without thinking)?
• Concerns: If there are hundreds/thousands of subbed channels, wouldn't it take too long to parse all of them and then fetch the latest n videos and their thumbnails (not CPU time, more concerned about net speed)?

@pystardust Have you started working on this already? May I know how you want to implement this? (asking cuz I want to be more involved in this wonderful project. I hope you don't mind)

[pystardust]

@cybarspace I would love people to be more involved. I have just thought about the idea. Currently im busy due to exams so havn't started it yet. As you mentioned having a file for subscriptions. Which contains channels' latest page urls and maybe a nick name for reference.

When say -S flag for subscriptions is called it will scrape the channel page just like this one scrapes the search page. I have checked out the json for channel page, and its similar to the search page but not the same. So we would need another scrape function.

Yes, it has a big speed drawback as it is technically pinging n number of searches for n channels.

Also as @JojiiOfficial has mentioned, we can cache the thumbnails.

[windowswithoutborders]

Just to throw in my two cents: I personally would be perfectly fine with a such a feature that, when invoked, brought up the channels.txt in fzf, and your selection from there would allow you to explore the channels uploads sorted from newest to oldest. This implementation might help when it comes to resources as the user can avoid pinging channels that may not upload frequently but they know they'd like to visit occasionally. Additionally, on invocation of such a feature have ytfzf parse a directory, ~/.config/ytfzf/channels/, for text files, where each text file is a list of channels as organized by the user (e.g., by channel genre: music.txt, science.txt, tech.txt). If more than one exists, display the files to select in fzf, potentially reducing any speed drawbacks by narrowing down the channels shown.

[Randalix]

Great to see you all so enthusiastic about it! We can gain back control and show youtube's algorithm :middle_finger: Caching the subscriptions is a fantastic idea. Will make it super fast! But also adds another layer of complexity. What would be a good solution for this kind of database? sqlite? json?

[cybardev]

All this is making me think... Should we really stick to POSIX Shell script? The project might soon choke while trying to bite more than it can chew...

Since we are increasing dependencies anyway and adding feature after feature, maybe port the code base to a language with nice APIs that make the devs' work easier and for users to get more features?

The script is about 500 LoC already. If a port is to be make, it's better to do it before the codebase becomes too big to switch.

[cybardev]

What would be a good solution for this kind of database? sqlite? json?

I think JSON would be fine. It's simple and easy, and easier to read imo. We're already having to parse JSON, so it's probably easier to add support for a different style of JSON than a completely new database language. Just my opinion.

[Euro20179]

All this is making me think... Should we really stick to POSIX Shell script? The project might soon choke while trying to bite more than it can chew...

I think this script is meant to be highly portable without installing much so python would be an alternative but python (imo) is basically just bash with classes

[Randalix]

I think this script is meant to be highly portable without installing much so python would be an alternative but python (imo) is basically just bash with classes

Python is more easy/readable and aren't classes a wonderful thing for things like this.

[cybardev]

Just to throw in my two cents: I personally would be perfectly fine with a such a feature that, . . . , display the files to select in fzf, potentially reducing any speed drawbacks by narrowing down the channels shown.

Yeah, I think this is a nice way to implement it. To summarize:

• ytfzf, invoked with a certain flag (or no arguments), displays a menu that's a list of channels (no images here)
• the list is to be read from a local JSON file holding channel data
• the database must contain at least: name, channel ID (maybe lask known number of subs, but inaccurate, I dislike)
• when user selects a channel from the first menu, another list menu pops up with the latest videos of selected channel (scraped just now, not before this step, for performance)
• there could be support for a "back" key to return to the previous menu (list of channels) in case you changed your mind on what channel to watch

This will make it pretty much like the YouTube homepage, customized, but it'd be under our control instead of Google suggesting decade-old cat videos.

Regarding the separation of channels by "categories". It's certainly possible and I suggest it still be included in the channel JSON file.

We could have this kind of structure:

{
"name": "Channel X",
"channel-id": "agry28hdh3893jd",
"tags": ["science", "nature", "animals"],
} 

This way we can get the same channels under multiple tags, which is often appropriate as many channels don't just focus on one topic but a collection of related topics.

[Randalix]

Using tags is a great idea :+1:

[Euro20179]

Python is more easy/readable and aren't classes a wonderful thing for things like this.

Python could work I don't really care what language is used so I think it's mainly up to @pystardust on which language should be used as it's their script.

But also with classes I feel like you (or at least i) usually end up with this

class Video:
    def __init__(self, link):
          self.link = link

    play(self):
        #code

    download(self):
        #code

videos = [] #list of videos

Can't you just do this

#list of videos (seperated by \n)
videos="" 

play_video () {
    #code 
}

download_video () {
    #code
}

In my opinion the biggest difference between the python script and the sh script is the lack of arrays/lists

[cybardev]

I think this script is meant to be highly portable without installing much

Well, that was what the dev initially had in mind ig, but as it seems we're adding optional dependencies anyway for features. I think it'd be a great experience for all of us to make this app much more versatile and feature-rich for users, and maintainable for devs. And I think using a language with a few module dependencies will allow us to make that happen without too much trouble. Python is Marvelous. My fork of this project is the only one that uses Python I think, so I know it's possible.

Plus if we get enough users to like the program and can scale it big enough, we could maybe land a job at Google The Linux Foundation, for the young ones among us who are interested in that kind of thing (myself included). :3 I mean, wanting to build a portfolio isn't bad right? 😅

[Euro20179]

I mean, wanting to build a portfolio isn't bad right?

haha

But yeah, I think it's just up to pystardust

Edit: and i mean personally i've never contributed to a repo before, and I suck at git so this is more of a learning experience for me

[cybardev]

As for OOP... I personally don't like it too much for these kind of things. It's great when you need custom data structures, and I'm not quite sure how that is needed for this project. I think this script would go well with functional programming. Python is still fine for that imo. Lambdas, generators, maps, etc. And a scraper that I can actually understand (urllib request, parse).

As for the final decision, it is ofc upto the repo owner. But y'know, he does have "py" in his username... Maybe it's destiny~ :3

[cybardev]

personally i've never contributed to a repo before, and I suck at git so this is more of a learning experience for me

I've also never contributed to someone else's repo, nor worked with a team. Y'all are wonderful and I'd love to grow this project with you guys. 😊

[Euro20179]

he does have "py" in his username

haha yes

and I can see maybe like one or two uses for OOP here like Video class, Channel class, Comment class but like with my video class example i think it could be done in a functional way

[cybardev]

Video class, Channel class, Comment class but with my video class example i think it could be done in a functional way

Yeah, I mean it can be done ofc, but probably more efficient to use functional paradigm mostly. Probably could have a mixture of both functional and OOP, as seen to be appropriate. We could have channels as dicts too, thus going either fully Python or mostly Python, eliminating the need to use JSON (tho json can certainly still be read in Python, and may be the more scalable).

[Euro20179]

We could have channels as dicts too, thus going either fully Python or mostly Python, eliminating the need to use JSON

the json module (built into python) converts json strings to dictionary or vice versa so if there were a subscriptions file that's json this should work if i remember right

import json
with open("subscription_file", "r") as subscriptions_file:
    subscriptions = json.load(subscriptions_file)

[cybardev]

the json module (built into python) converts json strings to dictionary or vice versa so if there were a subscriptions file that's json

Yeah, using JSON would be the better option. I'm just imagining the possibilities... 😅

Also, now I think I see an appropriate and potentially very useful application of OOP in this project.

Menu superclass with these subclasses:

• channel menu
• tag menu
• media menu
• any menus/lists we might add in the future (like you said ig, list of scraped comments)

[Euro20179]

Menu superclass with these subclasses

I can't see the menus being that different to each need a different class but they definitely could be and it would allow for easier customization in the program itself or a config file

[cybardev]

I can't see the menus being that different to each need a different class

Yeah, menus could just be objects of a menu class with some added stuff for each type of menu. Making subclasses might be too much.

[pystardust]

@cybarspace I like the idea of adding many more features in the script. But it would be nice to keep the script posix. And make it such a way that any person could understand whats happening and modify it. Initially this started as a single line posix script. But then we extended it by making sure it works on posix complaint shells. I really didn't expect people to like the project so much. Since we already ask for jq as a dependency we can parse json. Moreover scraping the site is basically scraping the json from the html. Since we already have the json dictionary in the variable all we need to do is parse them in a desirable way. Arrays and structures can be replaced with appropriate text stream and formatting, like unix keeps all its files in the /proc or use jq again to store jsons.

Lastly this is my first major project and I have no much experience in git. I am not a computer student, I just felt posix was fun so started the project so that I can learn it better. Feel free to use the scraping algorithm and remake it another language.

[pystardust]

We can implement the youtube home page/ any page on youtube which doesn't need a log in. Subscriptions are the only issue, it won't take much time to load up all the subscription the way we scrape the site. We can run all the curls in background and wait. It would be like opening 2 search results in 2 different tabs simultaneiously. Since I have exams going on I will try implementing them later this weekend, but I would be happy to take in PRs if any one wants to give it a shot. You basically need to change the scrapy_yt function to adjust to a channel page. Since almost every thing related to selection is done, there wouldn't be much lines added to the code.

[cybardev]

I like the idea of adding many more features in the script. But it would be nice to keep the script posix. And make it such a way that any person could understand whats happening and modify it.

Sure, if you want to keep it POSIX that's fine too. Although, POSIX does make it difficult for enabling anyone to understand it. Even I don't know how the script really works...

Initially this started as a single line posix script. But then we extended it by making sure it works on posix complaint shells. I really didn't expect people to like the project so much.

Haha, yeah, I remember when it was just a little script. It blew up really fast. :grinning:

Since we already ask for jq as a dependency we can parse json. Moreover scraping the site is basically scraping the json from the html. Since we already have the json dictionary in the variable all we need to do is parse them in a desirable way. Arrays and structures can be replaced with appropriate text stream and formatting, like unix keeps all its files in the /proc or use jq again to store jsons.

This is the part that goes all over my head... Sorry... I'm just not too good with *nix tools. Still learning. If you see my little Python fork, I just use regexp to parse the JSON and get the videoID. I will try to look into how jq works to be more useful for the project.

Lastly this is my first major project and I have no much experience in git. I am not a computer student, I just felt posix was fun so started the project so that I can learn it better. Feel free to use the scraping algorithm and remake it another language.

No worries. This is my first time actually contributing too. I'm also learning git. Haven't used it for much other than small random repos.

As for the translation to another language, I think I'll try to do that. I think I'm better at Python than shell scripting, so I'll try to make a branch in my fork that is a direct translation of your script in Python. My only concern is how fast you are going :laughing: It'll be hard for me to keep up, but I'll try my best. Thanks.

Edit: ^ I'm actually not going to do that. I'll try to get fzf menu working with Python, but I won't aim for complete feature parity (thumbnails, subscriptions, etc.). I'll keep it simple.

I would love to work on subscriptions, but I don't actually want to risk modifying what I don't understand... Hopefully someone else can make a PR.

[pystardust]

Adding this to the TODO's, closing the issue