arrow-benchmarks-ci

Arrow Benchmarks CI is responsible for

• orchestrating and prioritizing benchmark builds for Apache Arrow based on the list of benchmarks in Benchmarks repo
• publishing benchmark results to Apache Arrow pull requests associated with commits to master branch (example)
• posting benchmarks results to Conbench

Arrow Benchmarks CI consists of

• Buildkite pipelines and scripts for running benchmarks on benchmark machines
• Arrow BCI API service responsible for
  • listening to apache/arrow Pull Request comment events so benchmark builds can be scheduled for pull requests with an @ursabot please benchmark comment
  • persisting benchmark build statistics (e.g., memory usage, run time, errors, and conda packages)

Arrow Benchmarks CI Public Buildkite Pipelines

• Arrow BCI Test is used for testing Arrow BCI API service and CI scripts

Note that you can view builds for these pipelines but you can not manually schedule new builds.

How can I add my own benchmark machine to Arrow Benchmarks CI

Benchmark machines should be bare metal machines dedicated to only running benchmarks to avoid high variability in benchmark results that can result in false regression/improvements.

Please use this doc to add a new benchmark machine: How to Add New Benchmark Machine

How can I test benchmark builds that run on ursa-i9-9960x and ursa-thinkcentre-m75q locally

# Set env vars for conbench credentials if you need benchmark results to be posted to Conbench during testing
export CONBENCH_EMAIL=<conbench_user_id>
export CONBENCH_PASSWORD=<conbench_user_password>

# Build docker image with Arrow dependencies installed
cd ~/arrow-benchmarks-ci
docker build -f buildkite/benchmark-test/Dockerfile . -t benchmark-test

# Run benchmarks
docker run -i \
    --env BENCHMARKABLE=${BENCHMARKABLE:-"ac2d8ff481816299e2b047bf8a4546baccc3d050"} \
    --env BENCHMARKABLE_TYPE=${BENCHMARKABLE_TYPE:-"arrow-commit"} \
    --env BENCHMARKS_DATA_DIR="/data" \
    --env CONBENCH_EMAIL=$CONBENCH_EMAIL \
    --env CONBENCH_PASSWORD="$CONBENCH_PASSWORD" \
    --env CONBENCH_URL="https://conbench.ursa.dev" \
    --env MACHINE="docker-container-for-testing-benchmark-builds" \
    --env PYTHON_VERSION=${PYTHON_VERSION:-"3.8"} \
    --env RUN_ID=$BUILDKITE_BUILD_ID \
    --env RUN_NAME=${RUN_NAME:-"benchmark build test: $BUILDKITE_BUILD_ID"} \
    benchmark-test bash buildkite/benchmark/utils.sh create_conda_env_and_run_benchmarks

How can I test benchmark builds that run on Buildkite machines (ursa-i9-9960x, ursa-thinkcentre-m75q, etc)

This option is only available to Apache Arrow CI Buildkite org users at the moment.

1. Go to https://buildkite.com/apache-arrow/arrow-bci-benchmark-build-test
2. Click New Build
   • Set Commit and Branch for arrow-benchmarks-ci repo's commit and branch that you would like to test
   • Click Create Build
   • Wait for Pipeline upload step to finish
   • Click on Test Benchmark Builds button
   • Select machines you want to run test builds on on
   • Click Continue
   • Once Test Benchmark Builds, you will be able to monitor scheduled builds for selected arrow-benchmarks-ci repo's commit and branch under machines' Arrow BCI Benchmark on ... pipelines.

How to run tests

cd ~/arrow-benchmarks-ci
docker-compose -f envs/test/docker-compose.yml down
docker-compose -f envs/test/docker-compose.yml build
docker-compose -f envs/test/docker-compose.yml run app pytest -vv tests/