search

PowerLoom / snapshotter-lite

https://powerloom.dev
Apache License 2.0
10 stars 10 forks source link

readme

Table of Contents

Overview

Snapshotter workflow

A snapshotter peer as part of Powerloom Protocol does exactly what the name suggests: It synchronizes with other snapshotter peers over a smart contract running on Powerloom Prost chain. It follows an architecture that is driven by state transitions which makes it easy to understand and modify.

Because of its decentralized nature, the snapshotter specification and its implementations share some powerful features that can adapt to your specific information requirements on blockchain applications:

Snapshotter Lite Node is a lightweight implementation of the Snapshotter Peer that can be used to build simpler use cases that do not require the full functionality of the Snapshotter Peer. Snapshotter Lite Node only has a single dependency on Python environment and hence has significantly lower resource requirements than the Snapshotter Peer. It is suitable for use cases where no aggregation is required and the data can be directly used from the base snapshots.

Architecturally, the Snapshotter Lite Node is similar to the Snapshotter Peer, it still has the same modular and highly configurable architecture, allowing for easy customization and seamless integration. It consists of three core components:

  1. Main Snapshotter Codebase:

    • This foundational component defines all the essential interfaces and handles a wide range of tasks, from listening to epoch release events to distributing tasks and managing snapshot submissions.

  2. Configuration Files:

    • Configuration files, located in the /config directory are linked to snapshotter-configs repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings.

  3. Compute Modules:

    • The heart of the system resides in the snapshotter/modules directory are linked to snapshotter-computes, where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types.

Snapshotter Architecture

The architecture has been designed to facilitate the seamless interchange of configuration and modules. To achieve this, we maintain these components in separate Git repositories, which are then integrated into the Snapshotter Peer using Git Submodules. As a result, adapting the system to different use cases is as straightforward as changing a Git branch, offering unparalleled flexibility and versatility.

For more information on using Git Submodules, please refer to the Git Submodules Documentation.

Features Unavailable in Snapshotter Lite Node

Snapshotter Lite Node is a lightweight implementation of the Snapshotter Peer and hence does not support the following features:

If you require any of these features, please consider using the Snapshotter Peer instead. If you are unsure about which implementation to use, please reach out to us on Discord and we will be happy to help you out.

Epoch Generation

An epoch denotes a range of block heights on the EVM-compatible data source blockchain, for eg Ethereum mainnet/Polygon PoS mainnet/testnet. This makes it easier to collect state transitions and snapshots of data on equally spaced block height intervals, as well as to support future work on other lightweight anchor proof mechanisms like Merkle proofs, succinct proofs, etc.

The size of an epoch is configurable. Let that be referred to as size(E)

Snapshot Generation

Processors as configured in config/projects.json calculate snapshots for each task type based on the filtering or any criteria defined for snapshot generation against this epochId which corresponds to collections of state observations and event logs between the blocks at height in the range [begin, end].

The project ID is ultimately generated in the following manner:

https://github.com/PowerLoom/snapshotter-lite/blob/79e251b72ae617c70555d9925964c38c4a80c09e/snapshotter/utils/snapshot_worker.py#L40-L60

The snapshots generated are the fundamental data models on which higher-order aggregates and richer data points are built (By Full Nodes)

For situations where data sources are constantly changing or numerous, it is impractical to maintain an extensive list of them. In such cases, specific data sources need not be defined explicitly. You can just define a filtering criteria for the data source contract address and the snapshotter will automatically generate snapshots for all data sources that match the criteria.

For complex use cases with a lot of data requirements, it is recommended to use the Snapshotter Peer instead of the Snapshotter Lite Node because it employs a more efficient and scalable architecture.

Snapshot Finalization

All snapshots per project reach consensus on the protocol state contract which results in a SnapshotFinalized event being triggered.

event SnapshotFinalized(uint256 indexed epochId, uint256 epochEnd, string projectId, string snapshotCid, uint256 timestamp);

Major Components

The Snapshotter Lite Node consists of the following major components:

System Event Detector

System Event Detector, defined in system_event_detector.py, is the main entry point for Snapshotter Lite Node. It tracks events being triggered on the protocol state contract running on the anchor chain and calls appropriate classes to handle them.

Events being tracked by the system event detector are:

Processor Distributor

The Processor Distributor, defined in processor_distributor.py, acts upon the events received from the System Event Detector and distributes the processing tasks to the appropriate snapshot processors. It is also responsible for acting on allSnapshottersUpdated, DailyTaskCompletedEvent and DayStartedEvent events to manage the snapshot generation process.

https://github.com/PowerLoom/snapshotter-lite/blob/79e251b72ae617c70555d9925964c38c4a80c09e/snapshotter/processor_distributor.py#L300-L345

RPC Helper

Extracting data from the blockchain state and generating the snapshot can be a complex task. The RpcHelper, defined in utils/rpc.py, has a bunch of helper functions to make this process easier. It handles all the retry and caching logic so that developers can focus on efficiently building their use cases.

Core API

This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in core_api.py.

The pooler-frontend that serves the Uniswap v2 dashboards hosted by the PowerLoom foundation on locations like https://uniswapv2.powerloom.io/ is a great example of a frontend specific web application that makes use of this API service.

Among many things, the core API allows you to access the finalized CID as well as its contents at a given epoch ID for a project.

The main endpoint implementations can be found as follows:

https://github.com/PowerLoom/snapshotter-lite/blob/79e251b72ae617c70555d9925964c38c4a80c09e/snapshotter/core_api.py#L237-L289

https://github.com/PowerLoom/snapshotter-lite/blob/79e251b72ae617c70555d9925964c38c4a80c09e/snapshotter/core_api.py#L293-L340

The first endpoint in GET /last_finalized_epoch/{project_id} returns the last finalized EpochId for a given project ID and the second one is GET /data/{epoch_id}/{project_id}/ which can be used to return the actual snapshot data for a given EpochId and ProjectId.

These endpoints along with the combination of a bunch of other helper endpoints present in Core API can be used to build powerful Dapps and dashboards.

You can observe the way it is used in pooler-frontend repo to fetch the dataset for the aggregate projects of top pairs trade volume and token reserves summary:

try {
      response = await axios.get(API_PREFIX+`/data/${epochInfo.epochId}/${top_pairs_7d_project_id}/`);
      console.log('got 7d top pairs', response.data);
      if (response.data) {
        for (let pair of response.data.pairs) {
          pairsData7d[pair.name] = pair;
        }
      } else {
        throw new Error(JSON.stringify(response.data));
      }
    }
    catch (e){
      console.error('7d top pairs', e);
    }

Setup

There are multiple ways to set up the Snapshotter Lite Node. You can either use the Docker image or run it directly on your local machine. However, it is recommended to use the Docker image as it is the easiest and most reliable way to set up the Snapshotter Lite Node.

Using Docker

  1. Install Docker on your machine. You can find the installation instructions for your operating system on the official Docker website.

  2. Clone this repository using the following command:

    git clone https://github.com/PowerLoom/snapshotter-lite powerloom

    This will clone the repository into a directory named powerloom.

  3. Change your working directory to the powerloom directory:

    cd powerloom

  4. Run build.sh to start the snapshotter lite node:

    ./build.sh

    If you're a developer and want to play around with the code, instead of running build.sh, you can run the following command to start the snapshotter lite node:

    ./build-dev.sh

  5. When prompted, enter $SOURCE_RPC_URL, SIGNER_ACCOUNT_ADDRESS, SIGNER_ACCOUNT_PRIVATE_KEY (only required for the first time), this will create a .env file in the root directory of the project.

  6. This should start your snapshotter node and you should see something like this in your terminal logs

    snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:17 | INFO | Current block: 2208370| {'module': 'EventDetector'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:18 | DEBUG | Set source chain block time to 12.0| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter enabled: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter slot is set to 1| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:20 | INFO | Snapshotter enabled: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 1|snapshotter-lite  | February 5, 2024 > 15:10:21 | INFO | Snapshotter active: True| {'module': 'ProcessDistributor'}
snapshotter-lite_1  | 0|core-api          | February 5, 2024 > 15:10:22 | INFO | 127.0.0.1:59776 - "GET /health HTTP/1.1" 200 | {}

  7. To stop the node, you can press Ctrl+C in the terminal where the node is running or docker-compose down in a new terminal window from the project directory.

NOTE: It is recommended to run build.sh in a screen or tmux session so that the process continues running even after you close the terminal.

Without Docker

If you want to run the Snapshotter Lite Node without Docker, you need to make sure that you have Git, and Python 3.10.13 installed on your machine. You can find the installation instructions for your operating system on the official Python website.

  1. Clone this repository using the following command:

    git clone https://github.com/PowerLoom/snapshotter-lite powerloom

    This will clone the repository into a directory named powerloom.

  2. Change your working directory to the powerloom directory:

    cd powerloom

  3. Run init.sh to start the snapshotter lite node:

    ./init.sh

  4. When prompted, enter $SOURCE_RPC_URL, SIGNER_ACCOUNT_ADDRESS, SIGNER_ACCOUNT_PRIVATE_KEY (only required for the first time), this will create a .env file in the root directory of the project.

  5. Your node should start in background and you should start seeing logs in your terminal.

  6. To stop the node, you can run pkill -f snapshotter in a new terminal window.

Monitoring and Debugging

Monitoring

TODO - Enter Node dashboard screenshots and information here

Debugging

Usually the easiest way to fix node related issues is to restart the node. If you're facing issues with the node, you can try going through the logs present in the logs directory. If you're unable to find the issue, you can reach out to us on Discord and we will be happy to help you out.

For Contributors

We use pre-commit hooks to ensure our code quality is maintained over time. For this contributors need to do a one-time setup by running the following commands.

Now, whenever you commit anything, it'll automatically check the files you've changed/edited for code quality issues and suggest improvements.

Case Studies

1. Pooler: Case study and extending this implementation

Pooler is a Uniswap specific implementation of what is known as a 'snapshotter' in the PowerLoom Protocol ecosystem. It synchronizes with other snapshotter peers over a smart contract running on the present version of the PowerLoom Protocol testnet. It follows an architecture that is driven by state transitions which makes it easy to understand and modify. This present release ultimately provide access to rich aggregates that can power a Uniswap v2 dashboard with the following data points:

Extending pooler with a Uniswap v2 data point

In this section, let us take a look at the data composition abilities of Pooler to build on the base snapshot being built that captures information on Uniswap trades.

Step 1. Review: Base snapshot extraction logic for trade information

Required reading:

As you can notice in config/projects.example.json, each project config needs to have the following components

There's currently no limitation on the number or type of usecases you can build using snapshotter. Just write the Processor class and pooler libraries will take care of the rest.

{
  "config": [{
      "project_type": "uniswap_pairContract_pair_total_reserves",
      "projects":[
        "0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc",
        "0xae461ca67b15dc8dc81ce7615e0320da1a9ab8d5",
        "0x0d4a11d5eeaac28ec3f61d100daf4d40471f1852",
        "0x3041cbd36888becc7bbcbc0045e3b1f144466f5f",
        "0xd3d2e2692501a5c9ca623199d38826e513033a17",
        "0xbb2b8038a1640196fbe3e38816f3e67cba72d940",
        "0xa478c2975ab1ea89e8196811f51a7b7ade33eb11"
        ],
      "processor":{
        "module": "pooler.modules.uniswapv2.pair_total_reserves",
        "class_name": "PairTotalReservesProcessor"
      }
    },
    {
      "project_type": "uniswap_pairContract_trade_volume",
      "projects":[
        "0xb4e16d0168e52d35cacd2c6185b44281ec28c9dc",
        "0xae461ca67b15dc8dc81ce7615e0320da1a9ab8d5",
        "0x0d4a11d5eeaac28ec3f61d100daf4d40471f1852",
        "0x3041cbd36888becc7bbcbc0045e3b1f144466f5f",
        "0xd3d2e2692501a5c9ca623199d38826e513033a17",
        "0xbb2b8038a1640196fbe3e38816f3e67cba72d940",
        "0xa478c2975ab1ea89e8196811f51a7b7ade33eb11"
        ],
        "processor":{
          "module": "pooler.modules.uniswapv2.trade_volume",
          "class_name": "TradeVolumeProcessor"
        }
    }
  ]
}

If we take a look at the TradeVolumeProcessor class present at snapshotter/modules/computes/trade_volume.py it implements the interface of GenericProcessorSnapshot defined in snapshotter/utils/callback_helpers.py.

There are a couple of important concepts here necessary to write your extraction logic:

Output format can be anything depending on the usecase requirements. Although it is recommended to use proper pydantic models to define the snapshot interface.

The resultant output model in this specific example is UniswapTradesSnapshot as defined in the Uniswap v2 specific modules directory: utils/models/message_models.py. This encapsulates state information captured by TradeVolumeProcessor between the block heights of the epoch: min_chain_height and max_chain_height.

Find us