Introduce business model - database layer

[MaksymZavershynskyi]

Currently we have the following issues:

• The lifetime of the business objects is managed by whatever entity they live in. E.g. block, state node, etc have their lifetime managed by high-level components of the system. Some business objects have a co-dependent lifetime, e.g. state and state changes are supposed to be computed simultaneously. We don't have a central place that enforces these assumptions on the dependencies of the lifetimes, instead this logic is scattered around our codebase, e.g. GC makes sure when it deletes blocks, states, and state changes that they are removed consistently; similarly runtime makes sure there is not state without state changes and NightshadeRuntime makes sure there is no state without the block. However, because this logic is scattered across our entire codebase we make mistakes. We already have an issue, currently due to state sync it is possible for state to exist without the corresponding state changes and it is not clear how RPC server will behave if requested these state changes.
• There is complex mapping between columns in kv-database and business objects, i.e. a single type of business object needs to use multiple columns, e.g. ColBlock, ColBlockHeight, ColBlockMisc. Also we many columns are used to created DB indices, like ColNextBlockWithNewChunk, ColReceiptIdToShardId, etc. Currently we have 38 columns and this mapping is not specified anywhere which makes it really hard to maintain;
• In many places in our codebase we clone business objects to avoid complexities with Rust lifetimes if we try avoiding cloning then we might end-up with resources shared across threads, i.e. Mutexes which are notoriously hard to maintain.

The proposed solution is to have an explicit layer that every component in our system would use to interface with the storage. The advantages:

• The main goal of the layer would be making sure the objects with co-dependent lifetimes are maintained correctly;
• It can be used to keep mapping of columns to business objects in one place, this is especially useful for columns that are used for indices;
• It will own all non-temporary business objects and the rest of our codebase will refer to them through some form of linking (e.g. Arc or Weak. Most of the objects living in ORM will be immutable);
• It can use arena-like allocators to significantly speed up our codebase (we have experimented with arena allocators in Rust in the past for TxFlow, and the difference is very noticeable).

For the record, this was already proposed one year ago: https://github.com/nearprotocol/nearcore/issues/575#issuecomment-463841872 .

1. 1. 1. Patterns to consider

• https://en.wikipedia.org/wiki/Object-relational_mapping (while we don't have relational database, ORM has long being used to denote layer between any non-necessarily relational database and business model objects);
• https://en.wikipedia.org/wiki/Entity_component_system

1. 1. 1. Examples of when it could benefit us

• Currently merkle tree of headers live separately from the headers, but they have the same lifetimes. E.g. merkle tree for header X exists in our system's memory or database only when the header X exists. Unfortunately it means we are duplicating code – every time we create/remove header we need to create/remove merkle tree. Instead the creation/removal of the header should automatically lead to the creation of the corresponding merkle tree/root. Specifically this code: should not live inside the client: https://github.com/nearprotocol/nearcore/pull/2643/files#diff-55d2aa3a5feb6934292d5ef48562522eR355

[MaksymZavershynskyi]

For the record. @evgenykuzyakov reported the following error:

Apr 01 22:40:16.205 ERROR sync: Cannot build sync header (get_state_response_header): DB Not Found Error: BLOCK: sWLzUyRwQeXiASrgZbSTvPaULCauqJBoZCX61aVGaVQ
 Cause: Unknown
 Backtrace: stack backtrace:
   0: failure::backtrace::Backtrace::new
   1: <near_chain::store::ChainStore as near_chain::store::ChainStoreAccess>::get_block
   2: <near_client::view_client::ViewClientActor as actix::handler::Handler<near_network::types::NetworkViewClientMessages>>::handle
   3: <actix::address::envelope::SyncEnvelopeProxy<A,M> as actix::address::envelope::EnvelopeProxy>::handle
   4: <actix::contextimpl::ContextFut<A,C> as core::future::future::Future>::poll
   5: tokio::task::raw::poll
   6: <std::future::GenFuture<T> as core::future::future::Future>::poll
   7: actix_rt::runtime::Runtime::block_on
   8: near::main
   9: std::rt::lang_start_internal::{{closure}}::{{closure}}
             at src/libstd/rt.rs:52
      std::sys_common::backtrace::__rust_begin_short_backtrace
             at src/libstd/sys_common/backtrace.rs:130
  10: main
  11: __libc_start_main
  12: _start

With this layer we would've been able to track what invariant was broken relatively easy.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.