neume-network / crawler

music NFT crawler
https://neume.network
GNU General Public License v3.0
12 stars 2 forks source link

add a SQL database to manage tracks #9

Closed il3ven closed 1 year ago

il3ven commented 1 year ago

This PR implements functions to manage tracks using a SQL database. The schema of the database is based upon the schema proposed in neume-network/schema#60.

We have Tracks which is a class that exposes functions such as upsertTrack, getTrack or getTracksChanged. getTracksChanged will be useful to consume neume's data or sync neume nodes.

The communication with the SQL database is happening through Knex. This means we can swap SQLite with something more production-ready such as PostgresSQL. The current recommended method to use neume in production is to sync the DB of your choice with a neume node. If the neume node is using Postgres the node can directly be used to serve users in production.

Also worth noting, that as discussed this implementation only maintains the latest state in SQL. I have kept the LevelDB log which can aid in maintaining intermediate state but it hasn't been tested. We can look into it later.