L2-Technology / sensei

A lightning node implementation for everyone
https://l2.technology/sensei
Other
199 stars 39 forks source link

rewrite database using seaORM to support multiple backends #64

Closed johncantrell97 closed 2 years ago

johncantrell97 commented 2 years ago

Note: This is a breaking change and all nodes created before this will need to be recreated.

It completely deletes the entire database directory and implementation.
It uses seaORM to add support for sqlite, postgres, and mysql database backends. It re-implements bdk::database::Database using this same database backend. 100% of the data related to Sensei (all nodes, bdk-based on-chain wallets, and ldk-based objects) is stored in a single database.

The only things left on the filesystem at this point are the config file, macaroons, and log files.

It also includes a MAJOR bugfix around multiple instances of parts of a node getting started (peer manager, chain sync, etc) for a single node.

johncantrell97 commented 2 years ago

It also introduces two new sub-crates

entity -- defines all the data models used in Sensei. Implements all the basic CRUD functionality and some useful conversion utilities. Theoretically you could use this sub-crate outside of Sensei for some other project that wanted to use the same database models.

migration -- defines all the migrations needed to bring your database up to the current schema. This will be useful going forward for providing an upgrade path for users on old versions of Sensei.

johncantrell97 commented 2 years ago

closes #55, #25, and #18

dunxen commented 2 years ago

Wondering if there is an easy way to split this up into maybe 3 PRs for easier review 🙂:

  1. entity crate
  2. migration crate
  3. Database rewrite
johncantrell97 commented 2 years ago

Wondering if there is an easy way to split this up into maybe 3 PRs for easier review 🙂:

  1. entity crate
  2. migration crate
  3. Database rewrite

Heh I'm not sure that would help that much since those first two PRs changes are all self-contained in their own directories anyway.

I'm also not too concerned with it being reviewed at this point since it's a completely breaking change. It might as well be the "initial commit" to the project.

johncantrell97 commented 2 years ago

Ok added mysql + postgres docker-compose file for easy local development/testing.

Ran into a few issues testing with mysql/postgres that needed to be addresses to keep the code completely database agnostic.

1) Integer types need to remain signed. Which is really annoying since all of our integer types are always unsigned. Luckily it should always be safe to try_into().unwrap() when going from unsigned to signed since the numbers will fit. Going the other direction you can just cast as without a problem.

2) Originally I was storing txid and script directly as binary BLOB types. However, there needs to be indexes on these columns for certain lookups required by bdk. Works fine on postgres/sqlite but mysql doesn't like BLOB indices that don't have a defined length. To keep everyone happy I am now storing these as hex strings. Adds an extra serialization/deserialization step but it works.

johncantrell97 commented 2 years ago

Updated README with some examples of how to connect with different database backends.

I have ran through all my smoke tests (need to automate these with cypress) on all 3 back-ends and am not running into any issues.

I think I'll merge this pretty soon.