Ballista: Distributed SQL Query Engine, built on Apache Arrow
Ballista is a distributed SQL query engine powered by the Rust implementation of Apache Arrow and Apache Arrow DataFusion.
If you are looking for documentation for a released version of Ballista, please refer to the Ballista User Guide.
Overview
Ballista implements a similar design to Apache Spark (particularly Spark SQL), but there are some key differences:
- The choice of Rust as the main execution language avoids the overhead of GC pauses and results in deterministic processing times.
- Ballista is designed from the ground up to use columnar data, enabling a number of efficiencies such as vectorized processing (SIMD) and efficient compression. Although Spark does have some columnar support, it is still largely row-based today.
- The combination of Rust and Arrow provides excellent memory efficiency and memory usage can be 5x - 10x lower than Apache Spark in some cases, which means that more processing can fit on a single node, reducing the overhead of distributed compute.
- The use of Apache Arrow as the memory model and network protocol means that data can be exchanged efficiently between executors using the Flight Protocol, and between clients and schedulers/executors using the Flight SQL Protocol
Architecture
A Ballista cluster consists of one or more scheduler processes and one or more executor processes. These processes can be run as native binaries and are also available as Docker Images, which can be easily deployed with Docker Compose or Kubernetes.
The following diagram shows the interaction between clients and the scheduler for submitting jobs, and the interaction between the executor(s) and the scheduler for fetching tasks and reporting task status.
See the architecture guide for more details.
Features
- Supports HDFS as well as cloud object stores. S3 is supported today and GCS and Azure support is planned.
- DataFrame and SQL APIs available from Python and Rust.
- Clients can connect to a Ballista cluster using Flight SQL.
- JDBC support via Arrow Flight SQL JDBC Driver
- Scheduler web interface and REST UI for monitoring query progress and viewing query plans and metrics.
- Support for Docker, Docker Compose, and Kubernetes deployment, as well as manual deployment on bare metal.
Performance
We run some simple benchmarks comparing Ballista with Apache Spark to track progress with performance optimizations. These are benchmarks derived from TPC-H and not official TPC-H benchmarks. These results are from running individual queries at scale factor 10 (10 GB) on a single node with a single executor and 24 concurrent tasks.
The tracking issue for improving these results is #339.
Getting Started
The easiest way to get started is to run one of the standalone or distributed examples. After that, refer to the Getting Started Guide.
Project Status
Ballista supports a wide range of SQL, including CTEs, Joins, and Subqueries and can execute complex queries at scale.
Refer to the DataFusion SQL Reference for more information on supported SQL.
Ballista is maturing quickly and is now working towards being production ready. See the roadmap for more details.
Contribution Guide
Please see the Contribution Guide for information about contributing to Ballista.