MongoDB - long term storage

[bobinson]

We have introduced a Redis in memory cache as we are not using a ingestion frame work or a event processing framework. In the step one we are doing a copy of the data to the MongoDB as per the dApp requirements.

This will have to be revisited and enhanced to a proper ingestion frame work at a later stage.

At this point in time Python requests will mostly handle our needs with the possibility of using Pandas.

Reference document: https://github.com/Ucen-Blockchain/streamplay/issues/3#issuecomment-458870210

• ING1 and ING2 are to be implemented.
• This should be package
• use the same repository with a proper naming / folder structure where we can take this out as a package at a later stage if needed.

3.1 use cases (ING)

• ING1 : read from in-memory cache and store the dato a data sink
• ING2 : the read and write performed over network should be designed to extend to other data sources and sinks - ie it the source could be other formats, the blockchain itself and the destination can be MongoDO, RethinkDB, Solr (ElasticSearch) etc. Make the design generic and extentible but for now lets consider Redis —> MongoDB

[bobinson]

Bobinson Bobby

Date: 10-05-2018

Version : 0.0003

Unlike traditional APIs which were created for financial transactions and depends on Proof of Work, UCEN blockchain is designed from ground zero to develop DApps. In addition to the blockchain with DPoS based consensus, UCEN blockchain also provides middle ware to develop DApps ranging from games, news portals, mobile applications, medical applications etc. With the changing landscape in data protection regulations, extreme care is taken to adapt to the changing scenario of data protection and privacy regulations.

Ucen blockchain's middle ware exposes Data services and APIs and micro services which can be used to develop enterprise class applications with INTERNET scale. The middle ware is horizondally scalable and independent of the blockchain scaling. This enables us to provide geographically distributed, scalable middle ware layer. We expect various DApps performing consuming the API and data end points performing transformations, visualizations and presentation based on the immutable transactions in the blockchain. The middle ware takes care of data ingestion from the blockchain, data sources, support for adding new ingestion destinations and sinks, caching, proxy services etc to build massively scalable applications on top of it.

Along with the middle ware direct access to the blockchain is also possible for wallet access, accessing the real time consensus state, transactions etc.

Middleware components

1. streaming
2. ingestion
3. micro-services

2 Streaming

Streaming of the blockchain is taking real time, over the network snapshots of the blockchian activities / transactions and storing them in a database which can be handled easily.

The streaming is further sub-divided into fetching the data and storing to a long term storage.

2.1 STR Blockchain ----> In-Memory-Cache

• (STR1) Design an algorithm to stream blockchain from the last irreversible block
• (STR2) The captured data must be stored in Redis
• (STR3) I/O failures must be handled
• (STR4) This must be a package that we can later call from a daemon or cli
• (STR5) Packaging must be done to enable PyPI publishing in the future
• (STR6) Pipfile based dependency handling
• (STR7) Multithreading/multiprocessing MUST be used only if needed or should be adequately justified
• (STR8) Python 3.6.4 and Ubuntu 16.0.4 MUST be supported
• (STR9) Usage of file system based files or locks should be done carefully as we will need passive logic which will scale horizontally
• (STR10) Transactions design patterns with init - run - commit should be used.
• (STR11) Thread safe dependencies must be used
• (STR30) Include tests in the package
• (STR31) Support for CI (jenkins ?)
• (STR41) deployment of new features - DevOps
• (STR 51) Support for Windows
• (STR52) Running as a service in windows and using powershell
• (STR60) Support for LSW - Linux Subsystem for Windows

2.2 Testing and QA - STR

STR3 - I/O failure scenarios

• network error
• crash of the micro service
• switching of multiple nodes possible ?
• Hard fork realted changes like consensus algorithm change.

3 Ingestion

In this section we will capture the logic for the micro service which will consume the data from the in-memory cache and store for long term storage. To decide on the data formats needed, we will consider the requirements from the dApps and implment required formats and transformations on a need basis.

The first step is to ensure the long term storage of the data. This layer will perform 2 different types of operations.

• Process the live streamed data
• Snapshot of historical data

3.1 handling live stream

The live streamed data is present in the memory cache and this is captured and stored to the MongoDB. The first version will be developed as a single process Python service.

3.2 Snaphot

The snapshot of the blockchain means taking copy of the database from a starting block to an ending block. Often this will be from the genesis block to the lastirresible block.

3.1 use cases (ING)

• ING1 : read from in-memory cache and store the dato a data sink
• ING2 : the read and write performed over network should be designed to extend to other data sources and sinks - ie it the source could be other formats, the blockchain itself and the destination can be MongoDO, RethinkDB, Solr (ElasticSearch) etc. Make the design generic and extentible but for now lets consider Redis —> MongoDB
• ING3 : Define target schema based on the requirement from the Game and Web. Caputture requirements and record this for the purpose.

[bobinson]

• sndp487 added a comment.May 13 2018, 1:10 AM Comment Actions

Q - Data stored in redis-server will grow continuously while streaming. How should MongoDB work along, two possible ways: Say 'p' does redis ---> mongoDB (1) p first stores all the data already present in redis cache and then updates on the go with every new block stored in redis. (2) p can run periodically, resuming from where it stopped last time.

[bobinson]

bobinson added a comment.May 14 2018, 10:39 AM Comment Actions

(2) p can run periodically, resuming from where it stopped last time.

there should not be any relation between storing data to MongoDB and Redis
blockchain --> redis should be independent of all other components of the system & so is P
p = will be dump system that was streamed from BC (blockchain ) to MongoDB.
Redis will have to take care of its needs like memory, CPU etc
any serialization, data transformation etc is not the job of "p"
What Redis does with the data is not concern of P.
if P gets an error, it will just report it
whatever is presented by Redis, P will take it.

At a later point we will apply transformations to do the data. We are taking this approach as right now its more important to get the data in MongoDB and then provide the APIs than anything else.