[Meta] Invidious and Kemal | Should Invidious switch web frameworks?

[syeopite]

This is a meta issue for discussing the future of Invidious' web framework.

Kemal has been with Invidious since it's very inception. From the first commit omarroth made, to the latest the community has done. It's been a constant throughout Invidious' history. However, this is where problems arise: Kemal is a microframework. It's far too simple. It doesn't even have support for a more structured routing.

We had to essentially implement our own full stack framework on top of Kemal:

• Authentication
• Database manipulation
• Jobs
• File handling
• Structured routing

and so on...

It's all things we had to create in house on top of Kemal. And at one point, we've got to ask ourselves, is it really worth it? Our own additions on top of Kemal increases the complexity of Invidious tenfold and makes everything almost impossible to maintain; most of our time is already spent catching up to YouTube, do we really want more code to deal with? Invidious and it's Kemal based infrastructure already experiences crashes, memory leaks, and so many missing features like migrations and automatic database creation.

If Invidious wants to improve, it shouldn't stick with Kemal. We need to switch to a proper full stack framework.

Of course, this would also requiring rewriting a massive portion of Invidious, not to mention the need to learn a new framework. Invidious could potentially get a slight performance hit or even increased build times too. It's all factors we have to take into account. Though, in the long run, I think switching frameworks could benefit Invidious greatly.

All in all, we really have to discuss if sticking with such a simple framework is a good idea.

[FranciscoPombal]

I am an outsider with virtually no stake as a contributor at the moment, but a very frequent user of Invidious instances.

That being said, here is my 2¢:

I consider myself well-informed, and someone who keeps up with the latest developments in computer and software engineering. Nevertheless, prior to actually looking at the source code of this project, I had never even heard about Crystal.

At first I thought "damn, the cool kids behind Internet reclamation projects are now using this Crystal language. Did I miss something?". However, recent data suggests that it is indeed still a very obscure language, not that I had been asleep at the wheel. It's popularity is close to COBOL and the "Want" is close to Perl (I do expect the rate of change in these metrics to be the reverse of those languages though :)).

But it's a big wide world after all, and even a "very obscure language" can already have a rather large ecosystem. After all, 0.56% of "a lot" is still "quite a lot". There are already many Crystal projects on GitHub other than Invidious with a lot of stars, as I've since found out.

Though I cannot advise on which frame work to switch to, due to my unfamiliarity with the language and its popular web frameworks, I would suggest sticking with whichever framework is the most popular and feature-rich. It is surely not as easy to find potential contributors and future maintainers for a Crystal project, than, say, something Python-based. So make use of all the community support and previous work that already exists so that prospecting contributors to Invidious can focus on Invidious, not on basic infrastructure for a language that they most likely don't already know (which is already a big barrier to entry).

Some quick research turned up https://github.com/isaced/crystal-web-framework-stars and https://crystalshards.org/shards/search?q=web+framework, which show that there are 2 web frameworks other than Kemal that are dominant in the Crystal ecosystem: https://github.com/amberframework/amber and https://github.com/luckyframework/lucky.

[89Q12]

I'm working on a prototype of invidious using lucky, just to give some insight on what works good and what not with lucky. As of now I haven’t got much done besides authentication and basic html layout for some views(popular, login and general layout: navbar, footer).

So far I see two things that could take time when switching to lucky(But that could also be just me not knowing enough about crystal/lucky): Jobs, working with user preferences but again that is probably just me not knowing enough.

Currently I don't have so much time to work on it but I plan to get it in a working state in the next 2-3 weeks with the important features of the invidious and maybe if I have enough motivation try the same with amber.

If you have any thoughts on what I'm doing let me know :)

[Rjevski]

Maybe an unpopular opinion considering all of the maintainers seem to be proficient in Crystal and (I am assuming) enjoy working with the language, but I would suggest considering switching languages entirely.

Crystal is IMO a significant barrier to entry when it comes to new contributors as it's a pretty obscure language and has little commercial usage (I'm being generous here - I don't actually know any company that uses it) meaning learning it is unlikely to be directly translatable to income, thus the only practical reason most people would learn the language is to contribute to Invidious which is a significant thing to ask for - I'd rather have the project direct all volunteer effort into keeping up with YouTube rather than wasting it on technicalities such as an obscure language/tooling they won't even be able to capitalize on commercially.

I am not proficient in Crystal (see reasons above), but does it even have a web framework that is maintained and has the necessary features? When I'm thinking of web framework I'm thinking of Rails, Laravel or Django - all solid options that already solved the majority of problems you'd encounter when creating a web application, have been battle-tested over a decade (or more) and have significant communities and commercial usage meaning they're not going away any time soon.

I also don't think Crystal's compiled nature is giving us much advantage here from a performance point of view - in case of a YouTube proxy the bottleneck is always going to be upstream rate-limits so the extra performance is of little to no benefit, and the static typed nature of it is nice but not necessarily mandatory; we're talking about a YouTube proxy here, not some avionics-grade software that must be correct, so losing static typing by switching to an interpreted language isn't a dealbreaker.

From a developer experience perspective - if you haven't noticed already - I'm not a huge fan of Crystal. IDE support is lacking compared to major languages such as Python, Rails or PHP and the compiled nature of it slows down development in a project where an iterative, trial-and-error approach is often required (it's not like YT is providing us a nice, documented API). I've also had major trouble getting it to compile MacOS and haven't tried again since. In contrast, running Python/Django projects (which is what I do for a living) is pretty much seamless.

Switching languages would also allow us to share efforts when it comes to keeping up with YouTube. youtube-dl is a Python package that can download from YT (and other media websites - it has a modular structure), so the burden of keeping up with YT can be shared (YTDL has a large community and userbase, so Invidious would benefit significantly) instead of duplicating efforts. Of course, the same can be achieved without switching the language of the entire project (instead only writing a thin wrapper around YTDL that exposes its functionality as an HTTP API, so it can be consumed from Invidious) but if everyone is onboard with switching to Python (and given that it would require a massive rewrite) then the existence of YTDL means we may not even need to rewrite some parts if we can just use YTDL's.

[SamantazFox]

@Rjevski To respond to the points you make

I would suggest considering switching languages entirely.

That has been tried, by @/cloudrac3r. The project is called newleaf and should have been a python rewrite of invidious (and funded by iv-org), but afaik, this goal has been abandonned (@TheFrenchGhosty and/or @Perflyst may know more, I arrived quite after that).

Crystal is IMO a significant barrier to entry when it comes to new contributors [...] thus the only practical reason most people would learn the language is to contribute to Invidious

To be really honnest, I learned Crystal in a week, mostly with the help of a Ruby dev. Crystal is extremely similar to Ruby, and I'm pretty sure that a Ruby dev could easily contribute without having to learn more than a few specifics.

I'd rather have the project direct all volunteer effort into keeping up with YouTube rather than wasting it on technicalities such as an obscure language/tooling they won't even be able to capitalize on commercially.

On that part, don't worry, we're often discussing with NewPipe and yt-dlp maintainers ^^ Useful info is shared, it's just implemented differently. And one of our biggest struggles at the time is the lsquic library, which needs to be updated.

I am not proficient in Crystal (see reasons above), but does it even have a web framework that is maintained and has the necessary features? When I'm thinking of web framework [...]

Yep, there is Amber and Lucky. Amber is marketed as "inspired by rails", but I've not dug enough in either of those to say if they're similar or not.

Though, yeah, I totally get that there are other battle tested alternatives, but as I said earlier, a rewrite is expensive in terms of time, and nobody has yet been able to do it.

I also don't think Crystal's compiled nature is giving us much advantage here from a performance point of view - in case of a YouTube proxy the bottleneck is always going to be upstream rate-limits [...]

The bottleneck can actually be the instance. We are caching (maybe quite aggressively) different kinds of data in the database to avoid requesting too often the youtube servers. This has the side effect of decreasing response latency for large channels (like LinusTechTips, PewDiePie, etc...) because we serve cached data instead of being a simple proxy.

Furthermore, large instances receive a ton of requests (yewtu.be floats in the neighborhood of 40M daily requests), so performance definitely matters (especially in order to run on smaller servers).

IDE support is lacking compared to major languages [...]

I've honnestly been surprised to see that there is a nice plugin for SublimeText 3! But yeah, other IDEs may not have that privilege...

the compiled nature of it slows down development in a project where an iterative, trial-and-error approach is often required (it's not like YT is providing us a nice, documented API)

I'm not sure what you mean by that. It is perfectly possible to only compile a single page of source code to test out your changes, and this isn't particularly slower than something else (my old laptop achieves the compilation of a file like src/invidious/user/preferences.cr in less than 15s).

I've also had major trouble getting it to compile MacOS and haven't tried again since. In contrast, running Python/Django projects (which is what I do for a living) is pretty much seamless.

On that part, I definitely don't have the money for a MacOS device, nor the time to try set up a VM (if that's even possible). I'd really want to see invidious running on as many devices as possible! One of the culprits could well be lsquic again, though.

Switching languages would also allow us to share efforts when it comes to keeping up with YouTube. [...]

I'll refer to previous comments: we already share knowledge with other projects, and a rewrite in python has already been tried.

[SamantazFox]

Oh, and @11Tuvork28, I looked at your code and I have a question: Why is there a bunch of nodeJS things in it?

Nevermind, I found the answer by myself. It's included in the "jumpstart" template repository they provide. Though, I'm not sure what it's meant for? Is it client side or server side JS?

[89Q12]

~Oh, and @11Tuvork28, I looked at your code and I have a question: Why is there a bunch of nodeJS things in it?~

Nevermind, I found the answer by myself. It's included in the "jumpstart" template repository they provide. Though, I'm not sure what it's meant for? Is it client side or server side JS?

Well it's just to package the client side js/css using Mix. Though I'm thinking about removing it at a later point in time as it's not really necessary. Anyways thanks for taking your time to look at the code :)

[matthewmcgarvey]

I think there's also an argument to be made for contributing back to kemal to extend it for our needs. The benefits would be that we don't have to figure out a way to migrate the app to new a framework, we get to help the most popular web framework in Crystal grow to support larger projects, and we still get the flexibility we want. The drawback would be that we wouldn't be able to "control" what Kemal looks like and it would be on a different project's timeline as to how the changes get rolled out, if at all. While I'm not sold on this way of doing things, I did make a first PR over on kemal starting the process https://github.com/kemalcr/kemal/pull/621.

While I've never personally used Sinatra, I was surprised to learn that it has built in ways of modularization that allow you to essentially re-create Rails if you need to, and I think that would be along the lines of what we'd have to add to kemal. Here's sinatra's documentation of the kind of thing I envision being added to Kemal: https://github.com/sinatra/sinatra#sinatrabase---middleware-libraries-and-modular-apps

[grkek]

The only solution to this problem is through Grip, it just works!

[cloudrac3r]

cc @Rjevski

That has been tried, by @/cloudrac3r. The project is called newleaf and should have been a python rewrite of invidious (and funded by iv-org), but afaik, this goal has been abandoned

Not quite the case. NewLeaf is a reimplementation of the Invidious API and CloudTube is a video player frontend for it, for pretty much the reasons you mentioned here. I wanted to focus on performance and stability since Invidious lacked those at the time (unsure if it still does but the situation seems better in general). I don't write Crystal and I wanted an alternative that others could extend (perhaps by writing their own frontend).

Some work on NewLeaf related to HTTP3 was briefly funded by iv-org during more uncertain times, but this is no longer the case.

NewLeaf and CloudTube are actually in a relatively good place, see https://tube.cadence.moe for a public instance. Pagination is missing but the essentials are here and I have been using it as my main YouTube frontend for the last 18 months. I am still working on CloudTube and NewLeaf now from time to time, and in fact I found this issue while looking for documentation on Invidious's API, and decided to comment on it :)

So if you wanted to see a similar thing in another language that's compatible with the same API, look at NewLeaf+CloudTube.

[github-actions[bot]]

This issue has been automatically marked as stale and will be closed in 30 days because it has not had recent activity and is much likely outdated. If you think this issue is still relevant and applicable, you just have to post a comment and it will be unmarked.