This project is a monorepo that contains the following packages:
@kadena-indexer/indexer
: The indexer package, which is responsible for scanning and storing blocks for Kadena blockchain.@kadena-indexer/terraform
: The Terraform configuration for provisioning the infrastructure required to run the indexer and the node.This project is configured to run in a dev container. You can use the Dev Containers: Open Folder in Container
command in VSCode to open the project in a dev container. This will automatically install the required dependencies and set up the environment. To use the dev container, you need to have Docker installed on your machine.
If you don't have Dev Containers installed, you can install it from the VSCode Marketplace.
Under the /terraform
directory, create an .env
file using the .env.template
as a reference and set the environment variables accordingly.
cp terraform/.env.template terraform/.env
TF_VAR_AWS_ACCESS_KEY_ID
is your AWS access key ID.
TF_VAR_AWS_SECRET_ACCESS_KEY
is your AWS secret access key.
TF_VAR_AWS_ACCOUNT_ID
is your AWS account ID.
TF_VAR_AWS_USER_NAME
is the name of the user you created in AWS.
TF_VAR_AWS_DB_USERNAME
is the username for the postgress database.
TF_VAR_AWS_DB_PASSWORD
is the password for the postgress database.
Under the /indexer
directory, create an .env
file using the .env.template
as a reference and set the environment variables accordingly.
cp indexer/.env.template indexer/.env
AWS_S3_REGION
is the region where the S3 bucket is located.
AWS_S3_BUCKET_NAME
is the name of the S3 bucket where the data will be stored.
AWS_ACCESS_KEY_ID
is the access key ID for the S3 bucket.
AWS_SECRET_ACCESS_KEY
is the secret access key for the S3 bucket.
SYNC_BASE_URL
is the base URL for the Kadena node.
SYNC_MIN_HEIGHT
is the minimum height to start syncing from.
SYNC_FETCH_INTERVAL_IN_BLOCKS
is the interval in blocks to fetch.
SYNC_TIME_BETWEEN_REQUESTS_IN_MS
is the time between requests in milliseconds.
SYNC_ATTEMPTS_MAX_RETRY
is the maximum number of attempts to retry.
SYNC_ATTEMPTS_INTERVAL_IN_MS
is the interval in milliseconds between attempts.
SYNC_NETWORK
is the network to sync.
DB_USERNAME
is the username for the postgress database.
DB_PASSWORD
is the password for the postgress database.
DB_NAME
is the name of the postgress database.
DB_HOST
is the host for the postgress database. You have the host after the resource creation, so you can check for this information in the AWS console or in terraform output (postgres_db_host).
Initialize your Terraform workspace, which will download the provider and initialize it with the values provided in the terraform.`tfvars`` file.
terraform init
Plan and apply the Terraform configuration to provision your AWS resources:
yarn terraform plan
yarn terraform apply
If you want to destroy the infrastructure created, you can use the following command:
yarn terraform destroy
Set up the indexer with the following commands:
yarn && yarn indexer build
Continuous process of streaming, headers, payloads and missing blocks from node to s3 bucket and from s3 bucket to database
yarn indexer dev:run
sudo docker build -t kadena-indexer:latest .
sudo docker run --env-file ./indexer/.env -p 3000:3000 kadena-indexer:latest
Scan for and store historical blocks.
yarn indexer dev:backfill
Listen for new blocks and store them in real-time.
yarn indexer dev:streaming
Scan for and store any blocks that were missed.
yarn indexer dev:missing
Start the header processing from S3 to the database.
yarn indexer dev:headers
Start the payload processing from S3 to the database.
yarn indexer dev:payloads
For testing workflows locally, act is required. Install it using Homebrew:
brew install act
If you want to run the terraform workflow manually, you can use the following command:
yarn run-terraform-workflow
If you want to run the indexer workflow manually, you can use the following command:
yarn run-indexer-workflow