Iteration 9: Make DLT Stack persistent

[gnulib]

Story

As a developer of applications built on top of DLT stack, I'd like the DLT stack to support persistent datastore for DB and world state, so that I can use my network node across reboots and process data at real world scale with persistent storage.

Acceptance Criteria

DLT stack should be enhanced to support following:

• support persistence, i.e., node restart should not loose world state
• app registration should not re-play seen transactions
• node should be able to process long running sessions with huge amount of transactions data

[gnulib]

Issues with in memory DB

Current unit tests use an in-memory DB for testing, which does not exactly behaves like a persistent DB (e.g., it does not checks if DB is already opened, if it was closed). This causes problems and introduces bugs when DLT stack instance is used against a real persistent DB ... e.g., opening an already open DB causes errors and locks.

Hence, we need to change our in-memory DB to check for DB open/close status, and return errors accordingly. This will force the unit tests to catch all the error conditions that we may be missing out currently.

[gnulib]

Locking

When running 10k+ transactions, DLT stack locks up intermittently. All new transaction submission locks up, whereas network layer continuous to perform. So, it seems out locking is not true "2-phase" locking (i.e. all locks are not getting acquired and released in same order). We either need to remove all lower level locks and only lock at the DLT stack layer (i.e. at all the execution entry points), or we need to implement correct ordering in lock acquisition for true "2-phase" locking.

Current locks are at following places:

• DLT stack
• world state
• sharder (disabled)
• DLT DB
• DEVp2p layer
• peer
• ...

Synchronizing multiple DB instances

~Current leveldb based implementation of persistent DB provider uses separate DB instance for each different DB namespace -- this is causing a problem when DB instances do not "flush" data synchronously. As a result, transactions are getting processed multiple times as following:~

• ~A transaction is saved to TxDB by endorser, it is expected that any future receipt of the same transaction will be discarded as duplicate~
• ~however, during shard sync, when same transaction is coming from multiple different peers in quick succession, TxDB instance does not show that a transaction exists already -- most likely because it did not flush from earlier save~
• ~however, world state comes from a different DB instance, which already has state updated from first time processing of the same transaction~
• ~therefore, when TxDB instance fails to detect already existing transaction (because it has not flushed yet), it is causing the same transaction to update world state again (since world state DB had already flushed earlier and is already up-to-date)~

While it's a good idea to have a single DB instance for all namespaces, however that is not the cause for transactions getting processed multiple times. A load test with in-mem DB also exhibited same issues, and root cause turned out to be how we process transactions during shard sync, as documented in #58

[gnulib]

Deadlock root cause analysis

When running a bulk load test, it was found that DLT stack gets into a deadlock state and stop processing network transactions or accept new transaction submissions. As an initial analysis, it was anticipated that lock ordering was not correct and hence stack was getting into deadlock due to cyclic locks.

As a resolution, removed all the lower level locks and gated all DLT stack processing with stack level locks. However, we continued to see deadlocks. As per the thread state of process when in deadlock, it was found that threads were waiting on semaphore locks -- which correspond to the channel queues.

This lead to conclusion that channel queue depth is too shallow for the amount of traffic/events being processed during the bulk load tests. Hence, increased the channel depth as a function of expected transactions throughput (similar to seen transaction cache), and that resolved the deadlock issue.

[gnulib]

Re-use world state DB connection

Currently, world state DB is closed after every transaction processing. This is a significant overhead when processing high volume of transactions (e.g. during shard sync, or during bulk load). Persistent DB should be able to re-user existing open DB connection multiple times.