NineChronicles DataProvider

NineChronicles.DataProvider is an off-chain service that stores NineChronicles game action data to a database mainly for game analysis.
Currently, this service only supports MySQL database.

Pre-requisite
Run
Development Guide
Current Table Descriptions
Migrating Past Chain Data to MySQL Database

Pre-requisite

MySQL and Entity Framework Core tools should be installed in the local machine.

Run

Before running the program, please refer to the option values in the latest official 9c-launcher-config.json and fill out the variables in appsettings.json.
In appsettings.json, AppProtocolVersionToken, StorePath, PeerStrings, MySqlConnectionString properties MUST be filled to run the program.
To setup the NineChronicles mainnet blockchain store to use in the StorePath, download and extract the latest mainnet snapshot to a desired location.
```
$ dotnet run --project ./NineChronicles.DataProvider.Executable/ -- 
```

Development Guide

This section lays out the steps in how to log a new action in the database.
The TransferAsset action is used as an example in this guide.

1. Setup Database

To setup the database, navigate to NineChronicles.DataProvider/NineChronicles.DataProvider.Executable directory on your terminal and run the following migration command.
```
dotnet ef database update -- [Connection String]
```
Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"

2. Create Model

In NineChronicles.DataProvider/NineChronicles.DataProvider/Store/Models directory, create a model file called TransferAssetModel.cs.

In general, TxId, BlockIndex, Date, and Timestamp are useful to add as default properties in a model because these values will help with query speed when table size increases.


namespace NineChronicles.DataProvider.Store.Models
{
using System;
using System.ComponentModel.DataAnnotations;

public class TransferAssetModel
{
    [Key]
    public string? TxId { get; set; }

    public long BlockIndex { get; set; }

    public string? Sender { get; set; }

    public string? Recipient { get; set; }

    public decimal Amount { get; set; }

    public DateOnly Date { get; set; }

    public DateTimeOffset TimeStamp { get; set; }
}
}


- In [NineChronicles.DataProvider/NineChronicles.DataProvider/Store/NineChroniclesContext.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/NineChroniclesContext.cs), add a DbSet called `TransferAssets` and its description for reference.

// Table for storing TransferAsset actions public DbSet TransferAssets => Set();


### 2. Create Store Method

- In [NineChronicles.DataProvider/NineChronicles.DataProvider/Store/MySqlStore.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/Store/MySqlStore.cs), add a following method that stores the `TransferAsset` data into MySQL.

public void StoreTransferAsset(TransferAssetModel model) { using NineChroniclesContext ctx = _dbContextFactory.CreateDbContext(); TransferAssetModel? prevModel = ctx.TransferAssets.FirstOrDefault(r => r.TxId == model.TxId); if (prevModel is null) { ctx.TransferAssets.Add(model); } else { prevModel.BlockIndex = model.BlockIndex; prevModel.Sender = model.Sender; prevModel.Recipient = model.Recipient; prevModel.Amount = model..Amount; prevModel.Date = model.Date; prevModel.TimeStamp = model.TimeStamp; ctx.TransferAssets.Update(prevModel); }

ctx.SaveChanges();

}


### 3. (Optional) Add Data Getter
In some cases, you need to handle state to get data to make model.  
To do this easily, you can make your own data getter inside [NineChronicles.DataProvider/DataRendering/](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider/DataRendering).  

### 4. Render & Store Action Data

- In [NineChronicles.DataProvider/NineChronicles.DataProvider/RenderSubscriber.cs](https://github.com/planetarium/NineChronicles.DataProvider/blob/development/NineChronicles.DataProvider/RenderSubscriber.cs), add a following render code

_actionRenderer.EveryRender() .Subscribe(ev => { try { if (ev.Exception is null && ev.Action is { } transferAsset) { var model = new TransferAssetModel() { TxId = transferAsset.TxId, BlockIndex = transferAsset.BlockIndex, Sender = transferAsset.Sender, Recipient = transferAsset.Recipient, Amount = Convert.ToDecimal(transferAsset.Amount.GetQuantityString()), Date = DateOnly.FromDateTime(_blockTimeOffset.DateTime), TimeStamp = _blockTimeOffset, }; MySqlStore.StoreTransferAsset(model); } } catch (Exception e) { Console.WriteLine(e); }


### 5. Add Database Migration

- Navigate to [NineChronicles.DataProvider/NineChronicles.DataProvider.Executable](https://github.com/planetarium/NineChronicles.DataProvider/tree/development/NineChronicles.DataProvider.Executable) directory on your terminal and run the following migration command

dotnet ef migrations add AddTransferAsset -- [Connection String]

Connection String example: "server=localhost;database=data_provider;port=3306;uid=root;pwd=root;"

Current Table Descriptions

Tables that NineChronicles.DataProvider stores data into are listed in NineChroniclesContext.cs.
Please refer to each DbSet's comment in NineChroniclesContext.cs for table descriptions.

Migrating Past Chain Data to MySQL Database

Migrate All Action Data

This command migrates all action data based on DataRendering to the designated MySQL database.
Options such as offset and limit are provided to specify which block data to migrate.

IMPORTANT) This migration tool requires you to have the necessary state data in the blocks you want to migrate (If your chain store lacks the state data, this tool will not work).


Usage: NineChronicles.DataProvider.Executable mysql-migration [--store-path <String>] [--mysql-server <String>] [--mysql-port <UInt32>] [--mysql-username <String>] [--mysql-password <String>] [--mysql-database <String>] [--offset <I
nt32>] [--limit <Int32>] [--help]

Migrate action data in rocksdb store to mysql db.

Options: -o, --store-path Rocksdb path to migrate. (Required) --mysql-server A hostname of MySQL server. (Required) --mysql-port A port of MySQL server. (Required) --mysql-username The name of MySQL user. (Required) --mysql-password The password of MySQL user. (Required) --mysql-database The name of MySQL database to use. (Required) --offset offset of block index (no entry will migrate from the genesis block). --limit limit of block count (no entry will migrate to the chain tip). -h, --help Show help message


### Migrate Battle Arena Ranking Data
- This command calculates the `battle arena` ranking of all participants at a specific block index (`migration-block-index`) and inserts the data to a designated mysql database.

Usage: NineChronicles.DataProvider.Executable battle-arena-ranking-migration [--store-path ] [--mysql-server ] [--mysql-port ] [--mysql-username ] [--mysql-password ] [--mysql-database <Stri ng>] [--migration-block-index ] [--help]

Migrate battle arena ranking data at a specific block index to a mysql database.

Options: -o, --store-path RocksDB store path to migrate. (Required) --mysql-server Hostname of MySQL server. (Required) --mysql-port Port of MySQL server. (Required) --mysql-username Name of MySQL user. (Required) --mysql-password Password of MySQL user. (Required) --mysql-database Name of MySQL database. (Required) --migration-block-index Block index to migrate. -h, --help Show help message


### Migrate User Staking Data
- This command calculates the `NCG staking amount` of all participants at a specific block index (`migration-block-index`) and inserts the data to a designated mysql database.
- Currently, the `slack-token` and `slack-channel` options for sending the data in `csv` format are required, however, these will be changed to optional in the near future.

Usage: NineChronicles.DataProvider.Executable user-staking-migration [--store-path ] [--mysql-server ] [--mysql-port ] [--mysql-username ] [--mysql-password ] [--mysql-database ] [-- slack-token ] [--slack-channel ] [--migration-block-index ] [--help]

Migrate staking amounts of users at a specific block index to a mysql database.

Options: -o, --store-path Rocksdb store path to migrate. (Required) --mysql-server Hostname of MySQL server. (Required) --mysql-port Port of MySQL server. (Required) --mysql-username Name of MySQL user. (Required) --mysql-password Password of MySQL user. (Required) --mysql-database Name of MySQL database to use. (Required) --slack-token slack token to send the migration data. (Required) --slack-channel slack channel that receives the migration data. (Required) --migration-block-index Block index to migrate. -h, --help Show help message

planetarium / NineChronicles.DataProvider

readme