Redis is capable of an outstanding amount of op/secs. Instead of using postgres as single datastore we should put redis in front of postgres ensuring a fair amount of operations per second.
Proposed solution
Redis being used to perform all simple read/write operations
Postgres being used as a readonly mechanism for APIs leveraging SQL
Postgres being used as a slow - sync later - datastore
What needs to be done for this to work
[ ] Implement a sync mechanism
[ ] When a new block is added to redis it should be added to postgres within a convenient timeframe
[ ] When a new block is removed it should be removed from pg too.
[ ] Writes should be only performed in a single point of code and only blocks and transactions should be written directly from code. (see below)
[ ] Account updates (balances, votes, etc) should be done using a postgres data trigger.
[ ] When node starts redis must be cleaned and restored using data coming from PG.
[ ] We should calculate the amount of memory needed to store the needed data in memory (Redis) and prompt the node operator if there is not enough ram available.
[ ] In memory data should not expire.
[ ] retry mechanism in case data saving fails.
[ ] automatic process exit if slow-storage is too desyncd from "fast-storage" (Forcing node to rollback to last version in db). NOTE: in sync mode slow-storage could easily be out-of-sync by several blocks. A timestamp mechanism should also be used.
[ ] produce tuned redis.conf
What are the benefits
Slow I/O separation from main logic.
postgres will become the ultimate source of truth.
atomic block saving easier to perform.
Better TPS value (see #121)
Partial achievement of CQRS.
Possibility to log all slow-storage write operations for better debugging.
Redis is capable of an outstanding amount of op/secs. Instead of using postgres as single datastore we should put redis in front of postgres ensuring a fair amount of operations per second.
Proposed solution
What needs to be done for this to work
What are the benefits
Edit: Not needed right now.