sierkov / daedalus-turbo

Daedalus Turbo: 10-times-quicker bulk synchronization for Cardano
Apache License 2.0
21 stars 3 forks source link

Contents

About

Daedalus Turbo is an open-source project that aims to improve drastically (>=10x) the blockchain synchronization performance of the Daedalus wallet, the primary fully decentralized wallet of the Cardano blockchain. The project has a two-year schedule presented in its roadmap, and its technical approach is based on two key ideas: reducing the necessary network bandwidth through the use of compression and maximizing the use of parallel computation during the processing of blockchain data. These ideas are further explained in the following research reports:

Features

Currently supported:

In active development:

As the project matures and moves through its roadmap, the list of supported features will grow.

Requirements

Test it yourself

One can test the software using two methods:

Each is described in more detail below.

Cardano Chang fork support

On September 1, 2024 Cardano has introduced a number of new features with its Chang hard fork. The support for them is being actively developed. However, for the time being, the previous builds of DT Explorer likely won't work. On the other hand, the command line demo described below should work but limit the data to the last epoch of Cardano's Babbage era.

Command line interface

Prerequisites

To test the command line interface, you need the following software packages installed:

Commands

Clone this repository and make it your working directory:

git clone https://github.com/sierkov/daedalus-turbo.git dt
cd dt

Build the test Docker container:

docker build -t dt -f Dockerfile.test .

Start the test container, with <cardano-dir> being the host's directory to store the blockchain data:

docker run -it --rm -v <cardano-dir>:/data/cardano dt

All the following commands are to be run within the container started by the previous command.

Download, validate, and prepare for querying a copy of the Cardano blockchain from a network of compressing proxies with entry points listed in etc/mainnet/turbo.json:

./dt sync-turbo /data/cardano

(Optional) Revalidate the data downloaded by sync-turbo for benchmark purposes:

./dt revalidate /data/cardano

(Optional) Compare the downloaded chain vs a Cardano Network node (relays-new.cardano-mainnet.iohk.io by default) and fetch differences if necessary:

./dt sync-p2p /data/cardano

Reconstruct the latest balance and transaction history of a stake key:

./dt stake-history /data/cardano stake1uxw70wgydj63u4faymujuunnu9w2976pfeh89lnqcw03pksulgcrg

Reconstruct the latest balance and transaction history of a payment key:

./dt pay-history /data/cardano addr1q86j2ywajjgswgg6a6j6rvf0kzhhrqlma7ucx0f2w0v7stuau7usgm94re2n6fhe9ee88c2u5ta5znnwwtlxpsulzrdqv6rmuj

Show information about a transaction:

./dt tx-info /data/cardano 357D47E9916B7FE949265F23120AEED873B35B97FB76B9410C323DDAB5B96D1A

Evaluate a Plutus script and show its result and costs:

./dt plutus-eval ../data/plutus/conformance/example/factorial/factorial.uplc

Pre-built binaries for Windows and Mac (ARM64)

The latest builds of the DT Explorer application can be found in the Assets section of the latest GitHub release page. It shows the new synchronization and history-reconstruction algorithms in a safe and easy-to-test way by working without private keys and directly reconstructing history of any payment and stake address.

How to install on Windows

Windows builds have been tested with Windows 11 (earlier versions may work but have yet to be be tested).

How to install on Mac

Mac builds have been tested with Mac OS Sonoma (earlier versions may work but have yet to be be tested).

How to use

Spread the word

Many in the Cardano community, including some developers of Daedalus, don't believe that it's possible to make it noticeably faster. This leads to a situation in which the development is not focused on its performance. If you're persuaded by the evidence presented here, share it on social media with those around you. Changing the beliefs of people can be harder than building top-notch technology. So, every single tweet and Facebook post makes a difference. Thank you!

Supporting network infrastructure

The high-speed delivery of blockchain data depends on a network of compressing proxies (Cardano nodes that can share their local chain with other nodes in a compressed format and which are needed until the support for compressed data transfers becomes part of the regular Cardano network protocol). The current network configuration is sufficient to support up to 100'000 high-speed (with a download speed of 200+ Mbps) full blockchain synchronizations per year and orders of magnitude more incremental ones. The plan how to scale the network capacity to support a billion clients is presented in the Scalability of Bulk Synchronization in the Cardano Blockchain paper.

Quality

The accuracy of the ledger state reconstruction has been tested by recreating the ledger state from raw blockchain data at the end (the presented method batches updates) of each post-Shelley epoch up to the mainnet's epoch 494, exporting it into the Cardano Node format, and comparing it with the snapshot produced by Cardano Node version 8.9.2. The source code of the tools used for the comparison is located in lib/dt/cli-test/test-node-state.cpp and lib/dt/cli-test/test-export-full.cpp

The indexing and history-reconstruction code has been tested using a sample of ten thousand randomly-selected stake keys. For 100% of those, the reconstructed ADA balance (excluding rewards) precisely matched the stake recorded in the ledger snapshot produced by Cardano Node. The testing was performed with slot number 106012751 at the tip of the blockchain. The code of the test is located in the lib/dt/cli-test/test-stake-balances.cpp file.

Roadmap

The development of the project is organized into the following milestones: Milestone ETA Status
M1: Show that transaction history can be reconstructed 10x quicker February 2023 Ready
M2: Analyze the scalability of the Cardano network protocol and prepare requirements for the accelerated one June 2023 Ready
M3: Full POC of Daedalus Turbo: fast blockchain data delivery and transaction-history reconstruction April 2024 In review
M4: Integrate the POC with the official Daedalus builds March 2025 In Progress

Due to the experimental nature of the project, the ETAs are tentative. The development can go both faster and slower than expected.

Benchmarks

The experiment directory contains the source code of benchmarks and experiments discussed in the research reports.

Compilation

The software is in its proof-of-concept stage, and only the build path with Docker described above is regularly tested. Nevertheless, compilation in other environments and with other compilers is possible and is tested from time to time. The below notes may be helpful if you decide to build the software outside of Docker.

Necessary software packages

Additionally on Windows:

Tested environments and compilers

Build instructions

Verify the presence of the necessary libraries and generate build files in cmake-build-release directory for a release build:

cmake -B cmake-build-release

Build dt binary using all available CPU cores (will be available in cmake-build-release directory):

cmake --build cmake-build-release -j -t dt