MVP: Enable Conduit Performance Testing

[yaovi-a]

Overview

The evolution of Indexer to Conduit introduced a modular solution for Algorand customers to be able to read and query information from the Algorand blockchain. Conduit allows Algorand users to create their own data pipelines to be able to query Algod information through the Indexer REST API layer.

Problem

Conduit introduces a new software product that should have a proper suite of automation tools that can track and monitor:

• Performance characteristics of Conduit on different hardware configurations
• Performance characteristics of Conduit's different Importer / Processor / Export combinations when reading and processing different types of transactions
• Performance enhancements or regressions to Conduit when new code is introduced
• Performance of conduit against blockchain transaction rate growth. How well does conduit keep up with Algod?

What we would like to measure: (all at an individual block level)

• Import times for different importers
• Processing times for different processors
• Export times for different exporters
• Plug-in specific values
  • Ex. Exporter - Postgres time spent generating SQL v Database response time

Goals

1. Dev Enablement: Enable Lamprey and Developers to properly create and destroy infrastructure with different hardware configurations such as machine types and storage volumes
2. Dev Enablement: Provide Lamprey and Developers IaC to quickly deploy Indexer/Conduit software to different infrastructure in an automated manner
3. Metrics and Monitoring: Provide Lamprey and Developers a method to monitor the progress of their tests and collect metrics on conduit processes

Outcomes

• Devs can develop, test, and measure indexer/conduit software without the overhead of setting up infrastructure to deploy their changes
• Devs can test different hardware configurations to measure performance of indexer/conduit
• Devs can collect metrics on indexer/conduit performance in an easy manner

MVP (Q1 / Q2 rollover?) Tasks:

• Provides devs with an automated method to deploy indexer/conduit software to different infrastructure
• Devs are trained to modify IaC files to deploy different infrastructure configurations
• Devs have an easy method to access to infrastructure that is running indexer/conduit software
• Devs have a way to be alerted on when their infrastructure + software is ready to be interacted with (i.e. when conduit is ready to be used)

Ideas

• A further automated version of setup-tools would be a good start
  • When refining consider how to simplify:
  • Provisioning / configuring different hardware configs
  • Configuring NVMe / Installation locations
  • Managing different versions of conduit (release and configuration)
  • Setting up the networking between different parts (conduit / algod / postgres)
  • Installing / configuring 3rd party software (postgres / prometheus / grafana)
  • Dashboard automation
• Getting the automation to a point where it is able to assist with configuring and deploying the tools could be a point where HF can hand it off.
• STRETCH: Provide extension points to add testing scripts (i.e "start", "round-x", "caught up")

In the long run, can set up on cluster to automate some things.

• EKS with software-defined storage (how are local NVMe drives provisioned in EKS?)

Can create a packer AMI to pre-configure a lot of this.

Send logs

• Fluentbit/filebeat agent, forward to OpenSearch

[urtho]

Hi there - I am experimenting with different HW configs - like moving indexes or account tables to PMEM TABLESPACE . If You have any synthetic workloads please share - I'll get back with results on different HW setups

[winder]

@urtho take a look at block-generator, it simulates an algod instance and creates blocks according to your specification for indexer to ingest.

https://github.com/algorand/indexer/tree/develop/cmd/block-generator