(R): Move Backend Rust

[VincentRPS]

Description

Using Python in our API is a limitation more than a gain. With our entire toolset already language agnostic, and with Rust having arguably a better experience to develop with than Python, this is a better time than ever to make the move.

With this move will come some several key factors:

• Moving Write Heavy Database processes (like ReadStates, Messages, and other such) to ScyllaDB This would be done on Rust since the Python wrapper for Cassandra/Scylla is utterly pure garbage and barely supports asyncio.
• Finalizing objects This would finalize the objects Derailed uses since we would most likely be using raw SQL (with sqlx) as apposed to an ORM (like diesel, or seaorm). Although it is in my comfort to use an ORM, especially in Rust because they are much faster than Pythonic ORMs.

And for those who were not listening more closely, we will not be removing PostgreSQL, or even replacing most of what we use it for currently. We are just going to be replacing tables which are write-heavy to Scylla, most data will stay on PostgreSQL because of relations and many other aspects like heavy duplication, which Scylla and Cassandra don't support very well. By heavy duplication, I mean something like guild_id in a member object, or other such.

Prototyping

For now, since these are such massive changes, we will be first sketching them out in a prototyping branch. This will serve as the foundations for the eventual Rustacean API, and will conclude this issue. The prototype will never be used in production, and will be used for testing frameworks and other such.

On the prototype, and even production version, nothing should fluke or change except for objects if necessary. This is to keep any wrappers and other such mostly unaffected by the change and keep them serving the Bots and Clients they do.

We will continue using Argon2, but continuing using itsdangerous is something that will have to be pondered over during prototyping.

Components

If applicable, or required, add the components which will be changed here. e.g. Gateway, API, etc.

• API
• Gateway (if objects are changed)

[VincentRPS]

Please note the prototype will not sync with any production changes as it's not needed

[VincentRPS]

I am gonna pause this project for a bit to focus on the client and more frontend related features

[AquaQuokka]

I am gonna pause this project for a bit to focus on the client and more frontend related features

Literally everyone be thinking:

FINALLY

[AquaQuokka]

Not me though. I know how important the backend is. Without the backend, literally nothing would work.

[VincentRPS]

I have decided this may be high priority. It will have to be implemented before we launch publicly, though not in our private betas.

Since our first private beta will have data completely wiped, it'd be a great chance to either write the Rust rewrite before or during the first private beta.

This so we don't have to write Scylla migrations, and so we don't have to worry about moving data. This is especially since, as mentioned in the issue, we may change some key parts of the SQL formula to better accomodate Derailed.

Do note, when any change DB-wise comes to a few critical models like Settings, Guilds, etc. The Gateway will have to be modified to fit the new schema changes.

But in short, this is something that needs to be done before anything public, and something which will be critical to Derailed.

Since we're already starting to add Docker Compose interop, it'd make it not much harder to deploy Derailed once this change comes into effect.

[VincentRPS]

The rust branch will become the new home to Derailed's newly becoming Rust code base.

The plan now is not to only rewrite the API but also the Gateway to make Derailed as reliable as possible.

Due to the state of ScyllaDB migrations and the state of the libraries around it did not satisfy me enough to enable the effort of building parts of Derailed on it, so we will, for the time being, only use PostgreSQL to simplify our systems.

Firstly though, and something I've not contemplated since Gateway v0 is message passing. Message passing is a core part of our systems and needs to be fast and functional.

We have two choices I have founded out throughout all of this. We could go back on our old foundations and use Redis, or RabbitMQ. This comes with the benefit of being able to easily send and receive messages in a distributed and scalable fashion. Though this is something I extensively covered the downsides of in my Gateway blog but in short:

• The bandwidth costs may extend as, unlike the Elixir Gateway or my next proposal, each gateway node receives every event instead of only the events the users connected need.
• Because of the above receiving events in the future may become slow even on Rust since a node could be receiving 1 million+ events per second.

Yet these two could theoretically be avoided if needed.

Now the second option is forming a customized system of exchange using gRPC and Tokio.

gRPC will be used to send an receive events sincerely. Like on Elixir, the Gateway will have a Guilds node which holds Tokio channels (not actually named channels, though easier to say: https://docs.rs/tokio/latest/tokio/sync/broadcast/index.html) specified to a Guild's ID on a hashmap. To receive events, clients simply connect to gRPC which sends them updates from a single Guild. Then to send events, the API calls gRPC which broadcasts it to clients. The Guilds service only store guilds in use and with clients connected.

Clients detect which node a Guild is in using consistent hashing and etcd. Simply, etcd updates the servers with Guilds attached and clients update their list. Guild nodes also try to detect this and delete offending Guilds from their hashmap, if any.

But that's it, long proposal and sorry for a bit too much detail but hopefully that explains my plan for the Gateway in depth. If you have any opinions or feedback on what I say, please leave a comment or message me and I'll try to respond intuitively.

But before I leave off a little more info:

• There will be a crate in the middle of the Gateway and API for database actions.
• Docker will also be support for the Gateway and APIs new Rust versions.
• The API will use actix(-web), with serde of course.
• The Gateway will use tokio-tungstensite, and check for permissions.
• The Gateway will use Tonic as a gRPC server and client.
• The new versions of these should theoretically not change most, if not any, of the current payloads, models, or tables.
• These changes will still break the previous version of Derailed.