search

pragmaxim-com / ergo-uexplorer

Supplementary ergo chain explorer/analyzer with Scala/ZIO
MIT License
15 stars 1 forks source link
blockchain cassandra ergo graphql scala

readme

Σrgo μΣxplorer

WIP - heavily developed (built on ZIO 2)

Rationale

Blockchain weak spot? Having R/W efficient, cryptographically provable state

Explorer weak spot? Indexing state efficiently and perform fast analytical queries

Solution

Supplementary, lightweight (μ = micro) Ergo explorer/analyzer with multiple modes of operation :

Properties :

Chain Indexer

Chain indexer syncs with Node and keeps polling blocks while discarding superseded forks. Valid "Source of Truth" transaction data downloaded from Nodes is persisted into cassandra within an hour. Secondary Data Views that are derived from Source of Truth can be easily reindexed very fast which allows for flexible development. Major Data Views derived from Source of Truth :

Requirements:

Plugin framework

A Jar with Plugin implementation can be put on classpath and it will be passed each new Tx from mempool or each new Block together with all instances necessary for analysis:

  def processMempoolTx(
    newTx: Transaction,                    // new Tx is passed to all plugins whenever it appears in mempool of connected Nodes
    utxoState: UtxoState,                  // Utxo State
    graph: Option[GraphTraversalSource]    // Janus Graph for executing arbitrary gremlin traversal queries
  ): Future[Unit]

  def processNewBlock(
    newBlock: Block,                       // new Block is passed to all plugins whenever it is applied to chain of connected Nodes
    utxoState: UtxoState,                  // Utxo State
    graph: Option[GraphTraversalSource]    // Janus Graph for executing arbitrary gremlin traversal queries
  ): Future[Unit]

See example modules/alert-plugin implementation which submits high-volume Txs and Blocks to Discord channel with detailed information. It was able to render entire graph of related transactions/addresses via passing Graphml to Retina, and providing hyper-link to Retina's UI interface but that was not resilient due to unpredictable graph size (seeking for alternative).

Graphql API

When ./start-all.sh script finishes, go to http://localhost:8085/playground and copy/paste the auth token from following snippet to HTTP HEADERS at bottom-left of the playground and follow documentation.

curl -L -X POST 'http://localhost:8081/v1/auth' \
  -H 'Content-Type: application/json' \
  --data-raw '{
    "username": "cassandra",
    "password": "cassandra"
}'

and copy {"authToken": "secret"} paste {"x-cassandra-token": "secret"}

Examples

There is an address bar in the playground UI where you select either DDL (/graphql-schema) or DML (/graphql/<keyspace>) :

DDL (/graphql-schema)

query {
  keyspace(name: "uexplorer") {
    node_outputs: table(name: "node_outputs") { columns { name } }
  }
}

Rest API

At https://localhost:8090/explorer/swagger

Rest-api-0 Rest-api-1

DML (/graphql/uexplorer)

query {
  node_outputs(filter: {header_id: {eq: "b0244dfc267baca974a4caee06120321562784303a8a688976ae56170e4d175b"}}){
    values {
      address
    }
  }
}
query {
  node_outputs(filter: {address: {eq: "88dhgzEuTXaVTz3coGyrAbJ7DNqH37vUMzpSe2vZaCEeBzA6K2nKTZ2JQJhEFgoWmrCQEQLyZNDYMby5"}}){
    values {
      box_id
    }
  }
}

Production Build

Not necessary as all docker images from docker-compose are publicly available :

$ docker build . -t pragmaxim/uexplorer-chain-indexer:latest
# tree
  ├── schema-tables.cql             # db schema applied at start-indexing.sh phase
  ├── schema-indexes.cql            # db indexes applied at start-querying.sh phase
  ├── ergo.conf                     # expects global env variable SCOREX_REST_API_KEY_HASH
  ├── chain-indexer.conf            # no need to change anything
  ├── docker-compose.cassandra.yml  # base docker-compose, nothing runs without cassandra
  ├── docker-compose.indexer.yml    # base for minimal indexing with locally running Ergo Node
  ├── docker-compose.node.yml       # apply to base for running also Ergo Node
  ├── docker-compose.stargate.yml   # apply to base for graphql querying
  ├── docker-compose.janusgraph.yml # apply to base for using apps like gremlin console or IDEs that interact with Janusgraph
  ├── start-indexing.sh             # starts indexing, feel free to start up services individually
  ├── start-querying.sh             # applies indexes and starts stargate for graphql querying
  ├── stop-all.sh                   # safely stops everything running
  └── drain-cassandra.sh            # cassandra must be stopped gracefully

Run

$ cd docker
$ start-indexing.sh
$ docker compose logs uexplorer-chain-indexer
11:43:19 Initiating indexing of 816 epochs ...
11:43:21 New epoch 0 detected, utxo count: 1025, non-empty-address count: 122, persisted Epochs: 1[0], blocks cache size (heights): 33[1 - 1538]
11:43:22 New epoch 1 detected, utxo count: 2049, non-empty-address count: 134, persisted Epochs: 2[0 - 1], blocks cache size (heights): 33[1025 - 2562]
11:43:23 New epoch 2 detected, utxo count: 3073, non-empty-address count: 235, persisted Epochs: 3[0 - 2], blocks cache size (heights): 33[2049 - 3586]
...
12:37:11 New epoch 815 detected, utxo count: 1878724, non-empty-address count: 158786, persisted Epochs: 816[0 - 815], blocks cache size (heights): 32[835585 - 835616]
12:37:45 New epoch 816 detected, utxo count: 1886283, non-empty-address count: 159562, persisted Epochs: 817[0 - 816], blocks cache size (heights): 32[836609 - 836640]

$ start-querying.sh

Development

If Ergo Node is not running on localhost, it will use peers in the network

$ sbt stage
$ cd modules/chain-indexer/target/universal/stage/
$ ./bin/chain-indexer

Troubleshooting:

Cleanup:

$ ./stop-all.sh
$ docker volume ls # cassandra and ergo volumes contain a lot of data

TODO