KBVE / kbve

KBVE Monorepo
https://kbve.com/
11 stars 12 forks source link

[Concept] : [Backend] : KBVE API - Refactor + ULID Migration #130

Closed h0lybyte closed 3 months ago

h0lybyte commented 11 months ago

Core Concept/Theory A clear and concise description of what the concept is. Ex. It would be cool if [...]

An overlayed API that will operate above the Appwrite / n8n instances, with the objective of monitoring and maintaining their data, executions, and extending outwards for additional applications.

The API was initialized 3 Days ago under -> https://github.com/KBVE/kbve/commit/4d582ff163dcb919d170a23e8577e85ffc727bc7

Now I am currently extending out the internal database and its relations. with the objective of having the API handle the data collection, refining, and processing.

The reason I wanted to split the database up was that we would want the API's database to be as clean and refined as possible. Moving logs, errors, history, etc.. into Grafana, thus keeping a large chunk of the API clean of information that does not need to be extended.

The API currently does not have a front end, as the majority of the interactions for the API should not involve any human behavior.

The schemas/code for the API are generated by Autogen, my part is only to ensure that they remain consistent and avoid any hallucinations.


Alternative Ideas Is there any other way this concept could be used?

This would solve some of the issues when n8n and appwrite have communication errors. Furthermore, we can deploy additional serverless functions outside of our network and use the results within our database.


Alternative Examples/Sources Are there any other references that you can provide?

No alternative examples as of right now.

Additional information Add any other context or examples of this concept here.

Will update as I add more information.

h0lybyte commented 9 months ago

The Rust Schemas as a reference point.

n8n


diesel::table! {
    n8n (id) {
        id -> Unsigned<Bigint>,
        uuid -> Unsigned<Bigint>,
        #[max_length = 256]
        webhook -> Varchar,
        #[max_length = 256]
        permissions -> Varchar,
        #[max_length = 256]
        keyhash -> Varchar,
        #[max_length = 256]
        label -> Varchar,
    }
}

Appwrite

diesel::table! {
    appwrite (id) {
        id -> Unsigned<Bigint>,
        uuid -> Unsigned<Bigint>,
        #[max_length = 256]
        appwrite_endpoint -> Varchar,
        #[max_length = 256]
        appwrite_projectid -> Varchar,
        #[max_length = 256]
        apppwrite_api_key -> Varchar,
        #[max_length = 64]
        version -> Varchar,
        created_at -> Timestamp,
    }
}
h0lybyte commented 9 months ago

Going to migrate from mysql to supabase, to refactor the backend database storage. This is mainly because of the recent announcement of fly.io providing managed supabase-postgres.

While I did enjoy the usage of mysql, I feel like Postgres might be a better move to standardize the data across multiple languages.

Going to postpone that migration and look at different avenue for the database vectors. I will update this ticket next week with more information.

h0lybyte commented 9 months ago

Going to clean up the /apps/rust_api_profile and begin converting the id/uuid to ulid within the /packages/kbve cargo library.

h0lybyte commented 8 months ago

I should be done with the refactoring and migration within the next couple hours.

h0lybyte commented 8 months ago

Migrating over the

hazardous_create_user to use ULID / Vec instead of the u64.

Reference to the older function: https://github.com/KBVE/kbve/blob/023fd8aef48b20c9e5cbfad1cc76a496c2dac88f/packages/kbve/src/playerdb.rs#L187

h0lybyte commented 8 months ago

The last function left for the migration is the auth_login_player, however, I am thinking of optimizing it into a single query to call, rather than the 3 queries that we were using before.

h0lybyte commented 7 months ago
h0lybyte commented 7 months ago

Three API Endpoints that we need to test case this month:

h0lybyte commented 3 months ago

Refactor is nearly done, just need to move parts of it into this repo.