search

tablelandnetwork / go-tableland

Go implementation of the Tableland database/validator - run your own node, handling on-chain events and serving read-queries
https://tableland.xyz/
MIT License
53 stars 10 forks source link
database golang sql sqlite tableland web3

readme

Tableland Validator

Review Test Go Reference Go Report Card License Release standard-readme compliant

Go implementation of the Tableland database—run your own node, handling on-chain mutating events and serving read-queries.

Table of Contents

Background

go-tableland is a Go language implementation of a Tableland node, enabling developers and service providers to run nodes on the Tableland network and host databases for web3 users and applications. Note that the Tableland protocol is currently in open beta, so node operators have the opportunity to be one of the early network adopters while the responsibilities of the validator will continue to change as the Tableland protocol evolves.

What is a validator?

Validators are the execution unit/actors of the protocol.

They have the following responsibilities:

In the future, validators will have more responsibilities in the network.

Validator and network relationship

The following diagram describes a high level interaction between the validator, EVM chains, and the external world:

To better understand the usual mechanics of the validator, let’s go through a typical use case where a user mints a table, adds data to the table, and reads from it:

  1. The user will mint a table (ERC721) from the Tableland Registry smart contract on a supported EVM chain.
  2. The Registry contract will emit a CreateTable event containing the CREATE TABLE statement as extra data.
  3. Validators will detect the new event and execute the CREATE TABLE statement.
  4. The user will call the mutate method in the Registry smart contract, with mutating statements such as INSERT INTO ..., UPDATE ..., DELETE FROM ..., etc.
  5. The Registry contract, as a result of that call, will emit a RunSQL event that contains the mutating SQL statement as extra data.
  6. The validators will detect the new event and execute the mutating query in the corresponding table, assuming the user has the right permissions (e.g., table ownership and/or smart contract defined access controls).
  7. The user can query the /query?statement=... REST endpoint of the validator to execute read-queries (e.g., SELECT * FROM ...), to see the materialized result of its interaction with the smart contract.

The description above is optimized to understand the general mechanics of the validator. Minting tables and executing mutating statements also imply more work both at the smart contract and validator levels (e.g., ACL enforcing), which are being omitted here for simplicity sake.

The validator detects the smart contract events using an EVM node API (e.g., geth node), which can be self-hosted or served by providers (e.g., Alchemy, Infura, etc).

If you're curious about Tableland network growth, eager to contribute, or interested in experimenting, we encourage you to try running a validator. To get started, follow the step-by-step instructions provided below. We appreciate your interest and welcome any questions or feedback you may have during the process; stay tuned for updates and developments in our Discord and Twitter.

For projects that want to use the validator API, Tableland maintains a public gateway that can be used to query the network.

Running a validator

Running a validator only involves running a single process. Since we use SQLite as the default database engine, it is embedded and has many advantages:

We provide everything you need to run a validator with a single command using a docker-compose setup. This will automatically build everything from the source code, making it platform-independent since most OSes support docker. The build process is also dockerized, so node operators don’t need to worry about installing compilers or similar.

If you like creating your own setup (e.g., run raw binaries, use systemd, k8, etc.), we’re also planning to automate versioned Docker images or compiled executables. If there are other setups you're interested in, feel free to let us know or even share your own setup.

The Docker Compose setup section below describes how to run a validator in more detail, including:

Reviewing this section is strongly recommended but not strictly necessary.

Usage

System requirements

Currently, we recommend running the validator on a machine that has at least:

Hardware requirements might change with time, but this setup is probably over provisioned in the current state. We’re planning to do a stress testing benchmark suite to understand and predict the behavior of the validator under different loads to have more data about potential future recommended system requirements.

Firewall configuration

If you’re behind a firewall, you should open ports :8080 or :443, depending on if you run with TLS certificates. By default, TLS is not required, thus, expecting :8080 to be open to the external world.

System prerequisites

There are two prerequisites for running a validator:

Tableland has two separate networks:

This guide will focus on running the validator in the mainnet network.

We do this for two reasons:

We’ll also explain how to run a validator using Alchemy as a provider for the EVM node API the validator will use. The configuration will be analogous if you use self-hosted nodes or other providers. Note that if you do want to support testnets, you can, generally, replace this documentation's mainnet reference with testnet (e.g., an environment variable with MAINNET would be TESTNET; docker/deployed/mainnet would shift to docker/deployed/testnet).

Install host-level dependencies

To run the provided docker-compose setup, you’ll need to have installed:

Note that there’s no need for a particular Go installation since binaries are compiled within a docker container containing the correct Go compiler versions. Despite not being strictly necessary, creating a separate user in the host is usually recommended to run the validator.

Create EVM node API keys

The current setup needs one API key per supported chain. The default setup expects Alchemy keys for the following: Ethereum, Optimism, Arbitrum One, and Polygon; QuickNode for Arbitrum Nova. But, you are free to use a self-hosted node or another provider that supports the targeted chains.

To get your Alchemy keys, create an Alchemy account, log in, and follow these steps:

  1. Create one app for each chain using the + Create App button.
  2. You’ll see one row per chain—click the View Key button and copy/save the API KEY.

To get your QuickNode Arbitrum Nova key, create a QuickNode account, log in, and follow these steps:

  1. Create an endpoint.
  2. Select Arbitrum Nova Mainnet.
  3. When you finish the wizard, you should be able to have access to your API key.

Note: For Filecoin, we recommend Glif.io RPC support, which does not require authentication; the .env variable's value (shown below) can be left empty.

Run the validator

Now that you have installed the host-level dependencies, have one wallet per chain, and provider (Alchemy, QuickNode, etc.) API keys, you’re ready to configure the validator and run it.

1. Clone the go-tableland repository

Navigate to the folder where you want to clone the repository and run:

git clone https://github.com/tablelandnetwork/go-tableland.git

Running the main branch should always be safe since it’s the exact code that the public validator is running. We recommend this approach since we’re moving quickly with features and improvements but expect soon to be better guided by official releases.

2. Configure your secrets in .env files

You must configure each EVM account's private keys and EVM node provider API keys into the validator secrets:

  1. Create a .env_validator file in docker/deployed/mainnet/api folder—an example is provided with .env_validator.example.

  2. Add the following to .env_validator (as noted, this focuses on mainnet configurations but could be generally replicated for testnet support):

    VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY=<your ethereum mainnet alchemy key>
VALIDATOR_ALCHEMY_OPTIMISM_MAINNET_API_KEY=<your optimism mainnet alchemy key>
VALIDATOR_ALCHEMY_ARBITRUM_MAINNET_API_KEY=<your arbitrum mainnet alchemy key>
VALIDATOR_ALCHEMY_POLYGON_MAINNET_API_KEY=<your polygon mainnet alchemy key>
VALIDATOR_QUICKNODE_ARBITRUM_NOVA_MAINNET_API_KEY=<your arbitrum nova mainnet quicknode key>
VALIDATOR_GLIF_FILECOIN_MAINNET_API_KEY=

    Note: there is also an optional METRICS_HUB_API_KEY variable; this can be left empty. It's a service (cmd/metricshub) that aggregates metrics like git summary and pushes them to centralized infrastructure (GCP Cloud Run) managed by the core team. If you'd like to have your validator push metrics to this hub, please reach out to the Tableland team, and we may make it available to you. However, this process will further be decentralized in the future and remove this dependency entirely.

  3. Tune the docker/deployed/mainnet/api/config.json :

    1. Change the ExternalURIPrefix configuration attribute into the DNS (or IP) where your validator will be serving external requests.

    2. In the Chains section, only leave the chains you’ll be running; remove any chain entries you do not wish to support.

      Reference: example entry ```json { "Name": "Ethereum Mainnet", "ChainID": 1, "Registry": { "EthEndpoint": "wss://eth-mainnet.g.alchemy.com/v2/${VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY}", "ContractAddress": "0x012969f7e3439a9B04025b5a049EB9BAD82A8C12" }, "EventFeed": { "ChainAPIBackoff": "15s", "NewBlockPollFreq": "10s", "MinBlockDepth": 1, "PersistEvents": true }, "EventProcessor": { "BlockFailedExecutionBackoff": "10s", "DedupExecutedTxns": true, "WebhookURL": "https://discord.com/api/webhooks/${VALIDATOR_DISCORD_WEBHOOK_ID}/${VALIDATOR_DISCORD_WEBHOOK_TOKEN}" }, "HashCalculationStep": 150 } ```

  4. Create a .env_grafana file in the docker/deployed/mainnet/grafana folder—an example is provided with .env_grafana.example.

  5. Add the following to .env_grafana:

GF_SECURITY_ADMIN_USER=<user name you'd like to login intro grafana>
GF_SECURITY_ADMIN_PASSWORD=<password of the user>

Note: the GF_SERVER_ROOT_URL variable is optional and can be left empty. By default, Grafana is hosted locally at http://localhost:3000.

That’s it...your validator is now configured!

It's worthwhile to review the config.json file to see how the environment variables configured in the .env files inject these secrets into the validator configuration. Also, note how supporting more chains only requires adding an extra entry in the Chains, so it's straightforward to add support for any of the supported testnets of each mainnet chain. Note that adding a new mainnet chain that's not yet supported by the network is not possible as this requires the core Tableland protocol to separately deploy a Registry smart contract in order to enable new chain support. This is performed on a case-by-case basis, so please reach out to the Tableland team if you'd like support for a new mainnet chain.

3. Run the validator

To run the validator, move to the docker folder and run the following:

make mainnet-up

Some general comments and tips:

The default docker-compose setup has a baked-in observability substack with Prometheus and Grafana. You can learn more about this in the next section.

While the validator is syncing, you might see the logs are generated rather quickly. In the docker/deployed/mainnet/api/database.db, you should expect that the SQLite database will start to grow in size.

Docker Compose setup

The docker-compose setup can feel a bit magical, so in this section, we’ll explain the setup's folder structure and important considerations. Remember that you don’t need to understand this section to run a validator, but knowing how things work is highly recommended.

Architecture and port bindings

When you run make mainnet-up, you’re running the following stack:

If you’re running the validator, you’ll see these four containers running with docker ps.

There’re two main port binding groups:

Regarding the containers:

Folder structure

The docker/deployed/mainnet folder contains one folder per process that it’s running:

From an operational point of view, you usually don’t have to touch these folders apart from the api/config.json or api/.env_validator if you want to change something about the validator configuration or secrets. The Prometheus setup has a default 15 days retention time for the time series data, so the database size should be automatically bounded.

Configuration files

The validator configuration is done via a JSON file located at deployed/mainnet/api/config.json.

This file contains general and chain-specific configuration, such as desired listening ports, gateway configuration, log level configuration, and chain-specific configuration, including name, chain ID, contract address, wallet private keys, and EVM node API endpoints.

The provided configurations in each deployed/<environment> already have everything needed for the environment and other recommended values. The environment variable expansion parts of the config.json file, such as secrets and other attributes in the .env_validator file, were explained in the secret configuration section above. For example, the VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY variable configured in .env_validator expands a ${VALIDATOR_ALCHEMY_ETHEREUM_MAINNET_API_KEY} present in the config.json file. If you want to use a self-hosted Ethereum mainnet node API or another provider, you can edit the config.json file in the EthEndpoint endpoint. This same logic applies to every possible configuration in the validator.

Observability stack

As mentioned earlier, the default docker-compose setup provides a fully configured observability stack by running Prometheus and Grafana.

This setup configures the scrape endpoints in Prometheus to pull metrics from the validator and data sources dashboard for Grafana. These automatically bound configuration files are in docker/observability/(grafana|prometheus) folders. They are not part of the state of the processes. This is intentional so that, for example, the dashboard is part of the go-tableland repository, and you’ll get automatic dashboard upgrades while is being improved or extended.

After you spin up the validator, you can go to http://localhost:3000 and access the Grafana setup. Recall that you configured the credentials in the .env_grafana file in docker/deployed/mainnet/grafana.

If you browse the existing dashboards, you should see an existing Validator dashboard that should look like the following, which aggregates all metrics that the validator generates:

Healthbot (optional)

The healthbot daemon is an optional feature of the docker-compose stack and is only needed if you support a testnet network; it's disabled by default.

The main goal of healthbot is to test e2e in order to see if the validator is running correctly:

In short, it tests most of the processing healthiness of the validator. For each of the target chains, you should mint a table with the following statement:

CREATE TABLE healthbot_{chainID} (counter INTEGER);

This would result in having four tables—one per chain:

You should create a file .env_healthbot in the docker/deployed/testnet/healthbot folder with the following content (an example is provided with .env_healthbot.example):

HEALTHBOT_ETHEREUM_SEPOLIA_TABLE=healthbot_11155111_{tableID}
HEALTHBOT_OPTIMISM_SEPOLIA_TABLE=healthbot_11155420_{tableID}
HEALTHBOT_ARBITRUM_SEPOLIA_TABLE=healthbot_421614_{tableID}
HEALTHBOT_FILECOIN_CALIBRATION_TABLE=healthbot_314159_{tableID}

Finally, edit the docker/deployed/testnet/healthbot/config.json file Target attribute with the public DNS where your validator is serving to the external world. This is the endpoint where the healthbot will be making the healthiness probes. Since running the healthbot requires custom tables to be minted, it’s disabled by default.

To enable running the healthbot, you should run the following make testnet-up with the HEALTHBOT_ENABLED=true environment value set:

HEALTHBOT_ENABLED=true make testnet-up

After a few minutes, you should see the HealthBot -e2e check section of the Grafana dashboard populated:

Pruning docker images (optional)

Removing old docker images from time to time may be beneficial to avoid unnecessary disk usage. You can set up a cron rule to do that automatically. For example, you could do the following:

  1. Run crontab -e.
  2. Add the rule: 0 0 * * FRI /usr/bin/docker system prune --volumes -f  >> /home/validator/cronrun 2>&1

Backups and other routines

All validators are equipped with a backup scheduler that runs a background routine that executes a backup process of the SQLite database file at a configurable regular frequency. Besides the main backup of the database, the Backuper process executes a VACUUM process in the backup file and compresses it with zstd.

How the backup process works

The backup process called Backuper takes a backup of SQLite database file and stores it in a local directory relative to where the database is stored.

The process uses the SQLite Backup API provided by mattn/go-sqlite3. It is a full backup in a single step. Right now, the database is small enough not to worry about locking and how long it takes, but an incremental backup approach may be needed when as the database grows in the future.

How the scheduler works

The scheduler ticks at a regular interval defined by the Frequency config. It is important to mention that the time it runs is relative to the epoch time. That means, as the validator becomes operational and healthy after a deployment, it will start a backup routine in the next timestamp multiple of Frequency relative to epoch. That allows having backup files evenly distributed according to timestamp.

Vacuum

After the backup is finished, it executes the VACUUM SQL statement in the backup database to remove any unused rows and reduce the database file. This process may take a while, but it's expected since there shouldn't be any other connections to the backup database at this point.

Compression

After vacuum, we shrink the database even further by compressing it using the zstd algorithm implemented by compress library.

Pruning

We don't keep all backup files around—at the end, we remove any files exceeding the backup's KeepFiles config, located in cmd/api/config.go. The default value is 5.

Filename convention

The backup files follow the pattern: tbl_backup_{{TIMESTAMP}}.db.zst. For example, it should resemble the following: tbl_backup_2022-08-25T20:00:00Z.db.zst.

Decompressing the file

If you're on Linux or Mac, you should have unzstd installed out of the box. For example, run unzstd tbl_backup_2022-08-25T20:00:00Z.db.zst (replace with your file name) to decompress the compressed database file.

Metrics

We collect the following metrics from the process through logs:

Timestamp.             time.Time
ElapsedTime            time.Duration
VacuumElapsedTime      time.Duration
CompressionElapsedTime time.Duration
Size                   int64
SizeAfterVacuum        int64
SizeAfterCompression   int64

Additionally, we collect the metric tableland.backup.last_execution through Open Telemetry and Prometheus.

Configs

The backup configuration files are located in the docker/deployed/mainnet/api/config.json file. The following is the default configuration:

"Backup" : {
  "Enabled": true,       // enables the backup scheduler to execute backups
  "Dir": "backups",      // where backup files are stored relative to db
  "Frequency": 240,      // backup frequency in minutes
  "EnableVacuum": true,
  "EnableCompression": true,
  "Pruning" : {
    "Enabled": true,  // enables pruning
    "KeepFiles": 5    // pruning keeps at most `KeepFiles` backup files
  }
}

Development

Get started by following the validator setup steps described above. From there, you can make changes to the codebase and run the validator locally. For a validator stack against a local Hardhat network, you can run the following from the docker folder:

For a validator stack against deployed staging environments, you can run:

Configuration

Note that for deployed environments, there are two relevant configuration files in each folder docker/deployed/<environment>:

Besides that, you may want to configure Grafana's admin_user and admin_password. To do that, configure the .env_grafana file with the values of the expected keys shown in .env_grafana.example. This all should have been set up already but is worth noting.

Contributing

PRs accepted. Feel free to get in touch by:

Small note: If editing the README, please conform to the standard-readme specification.

License

MIT AND Apache-2.0, © 2021-2024 Tableland Network Contributors