Mesh Bitcoin

MESH-BITCOIN IS CONSIDERED ALPHA SOFTWARE. USE AT YOUR OWN RISK.

This project is available open source under the terms of the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).

## Overview The `mesh-bitcoin` repository provides a reference implementation of the Mesh API for Bitcoin in Golang. This repository was created for developers of Bitcoin-like (a.k.a., UTXO) blockchains, who may find it easier to fork this reference implementation than write one from scratch. [Mesh](https://docs.cdp.coinbase.com/mesh/docs/welcome/) is an open-source specification and set of tools that makes integrating with blockchains simpler, faster, and more reliable. The Mesh API is specified in the [OpenAPI 3.0 format](https://www.openapis.org). Requests and responses can be crafted with auto-generated code using [Swagger Codegen](https://swagger.io/tools/swagger-codegen) or [OpenAPI Generator](https://openapi-generator.tech), are human-readable (easy to debug and understand), and can be used in servers and browsers. ## Features * Mesh API implementation (both Data API and Construction API) * UTXO cache for all accounts (accessible using the Mesh `/account/balance` API) * Stateless, offline, curve-based transaction construction from any SegWit-Bech32 Address * Automatically prune bitcoind while indexing blocks * Reduce sync time with concurrent block indexing * Use [Zstandard compression](https://github.com/facebook/zstd) to reduce the size of data stored on disk without needing to write a manual byte-level encoding ## System Requirements The `mesh-bitcoin` implementation has been tested on an [AWS c5.2xlarge instance](https://aws.amazon.com/ec2/instance-types/c5). This instance type has 8 vCPU and 16 GB of RAM. ## Getting Started 1. Adjust your [network settings](#network-settings) to the recommended connections. 2. Install and run Docker as directed in the [Deployment](#deployment) section below. 3. Run the [`Testnet:Online`](#testnetonline) command. ### Network Settings To increase the load that `mesh-bitcoin` can handle, we recommend tunning your OS settings to allow for more connections. On a linux-based OS, you can run these commands ([source](http://www.tweaked.io/guide/kernel)): ```text sysctl -w net.ipv4.tcp_tw_reuse=1 sysctl -w net.core.rmem_max=16777216 sysctl -w net.core.wmem_max=16777216 sysctl -w net.ipv4.tcp_max_syn_backlog=10000 sysctl -w net.core.somaxconn=10000 sysctl -p (when done) ``` _We have not tested `mesh-bitcoin` with `net.ipv4.tcp_tw_recycle` and do not recommend enabling it._ You should also modify your open file settings to `100000`. This can be done on a linux-based OS with the command: `ulimit -n 100000`. ### Memory-Mapped Files `mesh-bitcoin` uses [memory-mapped files](https://en.wikipedia.org/wiki/Memory-mapped_file) to persist data in the `indexer`. As a result, you **must** run `mesh-bitcoin` on a 64-bit architecture (the virtual address space easily exceeds 100s of GBs). If you receive a kernel OOM, you may need to increase the allocated size of swap space on your OS. There is a great tutorial for how to do this on Linux [here](https://linuxize.com/post/create-a-linux-swap-file/). ## Development While working on improvements to this repository, we recommend that you use these commands to check your code: * `make deps` to install dependencies * `make test` to run tests * `make lint` to lint the source code * `make salus` to check for security concerns * `make build-local` to build a Docker image from the local context * `make coverage-local` to generate a coverage report #### Image Installation Running these commands will create a Docker image called `mesh-bitcoin:latest`. ##### Installing from GitHub To download the pre-built Docker image from the latest release, run: ```text curl -sSfL https://raw.githubusercontent.com/coinbase/mesh-bitcoin/master/install.sh | sh -s ``` _Do not try to install mesh-bitcoin using GitHub Packages!_ ##### Installing from Source After cloning this repository, run: ```text make build-local ``` #### Run Docker Running these commands will start a Docker container in [detached mode](https://docs.docker.com/engine/reference/run/#detached--d) with a data directory at `/bitcoin-data` and the Mesh API accessible at port `8080`. ##### Required Arguments **`MODE`** - Determines whether Mesh can make outbound connections. - **Type:** `String` - **Options:** `ONLINE`, `OFFLINE` - **Default:** None **`NETWORK`** - The Ethereum network to launch or communicate with. - **Type:** `String` - **Options:** `MAINNET`, `ROPSTEN`, `RINKEBY`, `GOERLI` or `TESTNET` - **Default:** `ROPSTEN`, but only for backwards compatibility if you use `TESTNET` **`PORT`** - The port to use for Mesh. - **Type:** `Integer` - **Options:** `8080`, any compatible port number. - **Default:** None ##### Command Examples You can run these commands from the command line. If you cloned the repository, you can use the `make` commands shown after the examples. ###### **Mainnet:Online** Uncloned repo: ```text docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=MAINNET" -e "PORT=8080" -p 8080:8080 -p 8333:8333 mesh-bitcoin:latest ``` Cloned repo: ```text make run-mainnet-online ``` ###### **Mainnet:Offline** Uncloned repo: ```text docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=MAINNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest ``` Cloned repo: ```text make run-mainnet-offline ``` ###### **Testnet:Online** Uncloned repo: ```text docker run -d --rm --ulimit "nofile=100000:100000" -v "$(pwd)/bitcoin-data:/data" -e "MODE=ONLINE" -e "NETWORK=TESTNET" -e "PORT=8080" -p 8080:8080 -p 18333:18333 mesh-bitcoin:latest ``` Cloned repo: ```text make run-testnet-online ``` ###### **Testnet:Offline** Uncloned repo: ```text docker run -d --rm -e "MODE=OFFLINE" -e "NETWORK=TESTNET" -e "PORT=8081" -p 8081:8081 mesh-bitcoin:latest ``` Cloned repo: ```text make run-testnet-offline ``` ## Architecture `mesh-bitcoin` uses the `syncer`, `storage`, `parser`, and `server` package from [`mesh-sdk-go`](https://github.com/coinbase/mesh-sdk-go) instead of a new Bitcoin-specific implementation of packages of similar functionality. Below you can find an overview of how everything fits together: ### Concurrent Block Syncing To speed up indexing, `mesh-bitcoin` uses concurrent block processing with a "wait free" design (using [the channels function](https://golangdocs.com/channels-in-golang) instead of [the sleep function](https://pkg.go.dev/time#Sleep) to signal which threads are unblocked). This allows `mesh-bitcoin` to fetch multiple inputs from disk while it waits for inputs that appeared in recently processed blocks to save to disk. ## Test the Implementation with the mesh-cli Tool To validate `mesh-bitcoin`, [install `mesh-cli`](https://github.com/coinbase/mesh-cli#install) and run one of these commands: * `mesh-cli check:data --configuration-file mesh-cli-conf/testnet/config.json` - This command validates that the Data API information in the `testnet` network is correct. It also ensures that the implementation does not miss any balance-changing operations. * `mesh-cli check:construction --configuration-file mesh-cli-conf/testnet/config.json` - This command validates the blockchain’s construction, signing, and broadcasting. * `mesh-cli check:data --configuration-file mesh-cli-conf/mainnet/config.json` - This command validates that the Data API information in the `mainnet` network is correct. It also ensures that the implementation does not miss any balance-changing operations. Read the [How to Test your Mesh Implementation](https://docs.cdp.coinbase.com/mesh/docs/mesh-test/) documentation for additional details. ## Contributing You may contribute to the `mesh-bitcoin` project in various ways: * [Asking Questions](CONTRIBUTING.md/#asking-questions) * [Providing Feedback](CONTRIBUTING.md/#providing-feedback) * [Reporting Issues](CONTRIBUTING.md/#reporting-issues) Read our [Contributing](CONTRIBUTING.MD) documentation for more information. You can also find community implementations for a variety of blockchains in the [mesh-ecosystem](https://github.com/coinbase/mesh-ecosystem) repository. ## Documentation You can find the Mesh API documentation [here](https://docs.cdp.coinbase.com/mesh/docs/welcome/). Check out the [Getting Started](https://docs.cdp.coinbase.com/mesh/docs/getting-started/) section to start diving into Mesh. ## Related Projects * [mesh-sdk-go](https://github.com/coinbase/mesh-sdk-go) — The `mesh-sdk-go` SDK provides a collection of packages used for interaction with the Mesh API specification. * [mesh-specifications](https://github.com/coinbase/mesh-specifications) — Much of the SDK code is generated from this repository. * [mesh-cli](https://github.com/coinbase/mesh-ecosystem) — Use the `mesh-cli` tool to test your Mesh API implementation. The tool also provides the ability to look up block contents and account balances. ### Sample Implementations You can find community implementations for a variety of blockchains in the [mesh-ecosystem](https://github.com/coinbase/mesh-ecosystem) repository. ## License This project is available open source under the terms of the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0). © 2022 Coinbase