Databend: The Next-Gen Cloud [Data+AI] Analytics

Databend Serverless Cloud (beta) | Documentation | Benchmarking | Roadmap (v1.3)

🐋 Introduction

Databend, built in Rust, is an open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake. With its focus on fast query execution and data ingestion, it's designed for complex analysis of the world's largest datasets.

⚡ Performance

[TPC-H Benchmark: Databend Cloud vs. Snowflake](https://docs.databend.com/guides/benchmark/tpch)

Databend vs. Snowflake

[Data Ingestion Benchmark: Databend Cloud vs. Snowflake](https://docs.databend.com/guides/benchmark/data-ingest)

Databend vs. Snowflake

🚀 Why Databend

Cloud-Native: Integrates with AWS S3, Azure Blob, Google Cloud, and more.
High Performance: Rust-built, with cutting-edge, high-speed vectorized execution. 👉 ClickBench.
Cost-Effective: Designed for scalable storage and computation, reducing costs while enhancing performance. 👉 TPC-H.
AI-Powered Analytics: Enables advanced analytics with AI Functions.
Data Simplification: Streamlines data ingestion, no external ETL needed. 👉 Data Loading.
Real-Time CDC: Supports real-time incremental data updates to keep data current and accurate. 👉 Stream
Format Flexibility: Supports multiple data formats and types, including JSON, CSV, Parquet, GEO, and more.
ACID Transactions: Ensures data integrity with atomic, consistent, isolated, and durable operations.
Version Control: Provides Git-like version control for data, allowing querying, cloning, and reverting at any point.
Schemaless: VARIANT data type enabling schemaless data storage and flexible data modeling.
Flexible Indexing: Virtual Column, Aggregating Index, and Full-Text Index, for faster data retrieval.
Community-Driven: Join a welcoming community for a user-friendly cloud analytics experience.

📐 Architecture

Databend Architecture

🚀 Try Databend

1. Databend Serverless Cloud

The fastest way to try Databend, Databend Cloud

2. Install Databend from Docker

Prepare the image (once) from Docker Hub (this will download about 170 MB data):

docker pull datafuselabs/databend

To run Databend quickly:

docker run --net=host  datafuselabs/databend

🚀 Getting Started

Connecting to Databend

- [Connecting to Databend with BendSQL](https://docs.databend.com/guides/sql-clients/bendsql) - [Connecting to Databend with JDBC](https://docs.databend.com/guides/sql-clients/jdbc)

Data Import and Export

- [How to load Parquet file into a table](https://docs.databend.com/guides/load-data/load-semistructured/load-parquet) - [How to export a table to Parquet file](https://docs.databend.com/guides/unload-data/unload-parquet) - [How to load CSV file into a table](https://docs.databend.com/guides/load-data/load-semistructured/load-csv) - [How to export a table to CSV file](https://docs.databend.com/guides/unload-data/unload-csv) - [How to load TSV file into a table](https://docs.databend.com/guides/load-data/load-semistructured/load-tsv) - [How to export a table to TSV file](https://docs.databend.com/guides/unload-data/unload-tsv) - [How to load NDJSON file into a table](https://docs.databend.com/guides/load-data/load-semistructured/load-ndjson) - [How to export a table to NDJSON file](https://docs.databend.com/guides/unload-data/unload-ndjson) - [How to load ORC file into a table](https://docs.databend.com/guides/load-data/load-semistructured/load-orc)

Loading Data From Other Databases

- [How to Sync Full and Incremental MySQL Changes into Databend](https://docs.databend.com/guides/load-data/load-db/debezium) - [How to Sync Full and Incremental PostgreSQL Changes into Databend](https://docs.databend.com/guides/load-data/load-db/flink-cdc) - [How to Sync Full and Incremental Oracle Changes into Databend](https://docs.databend.com/guides/load-data/load-db/flink-cdc)

Querying Semi-structured Data

- [How to query directly on Parquet file](https://docs.databend.com/guides/load-data/transform/querying-parquet) - [How to query directly on CSV file](https://docs.databend.com/guides/load-data/transform/querying-csv) - [How to query directly on TSV file](https://docs.databend.com/guides/load-data/transform/querying-tsv) - [How to query directly on NDJSON file](https://docs.databend.com/guides/load-data/transform/querying-ndjson) - [How to query directly on ORC file](https://docs.databend.com/guides/load-data/transform/querying-orc)

Visualize Tools with Databend

- [Deepnote](https://docs.databend.com/guides/visualize/deepnote) - [Grafana](https://docs.databend.com/guides/visualize/grafana) - [Jupyter Notebook](https://docs.databend.com/guides/visualize/jupyter) - [Metabase](https://docs.databend.com/guides/visualize/metabase) - [MindsDB](https://docs.databend.com/guides/visualize/mindsdb) - [Redash](https://docs.databend.com/guides/visualize/redash) - [Superset](https://docs.databend.com/guides/visualize/superset) - [Tableau](https://docs.databend.com/guides/visualize/tableau)

Managing Users

- [How to Create a User](https://docs.databend.com/sql/sql-commands/ddl/user/user-create-user) - [How to Grant Privileges to a User](https://docs.databend.com/sql/sql-commands/ddl/user/grant#granting-privileges) - [How to Revoke Privileges from a User](https://docs.databend.com/sql/sql-commands/ddl/user/revoke#revoking-privileges) - [How to Create a Role](https://docs.databend.com/sql/sql-commands/ddl/user/user-create-role) - [How to Grant Privileges to a Role](https://docs.databend.com/sql/sql-commands/ddl/user/grant#granting-role) - [How to Grant Role to a User](https://docs.databend.com/sql/sql-commands/ddl/user/grant) - [How to Revoke the Role of a User](https://docs.databend.com/sql/sql-commands/ddl/user/revoke#revoking-role)

Managing Databases

- [How to Create a Database](https://docs.databend.com/sql/sql-commands/ddl/database/ddl-create-database) - [How to Drop a Database](https://docs.databend.com/sql/sql-commands/ddl/database/ddl-drop-database)

Managing Tables

- [How to Create a Table](https://docs.databend.com/sql/sql-commands/ddl/table/ddl-create-table) - [How to Drop a Table](https://docs.databend.com/sql/sql-commands/ddl/table/ddl-drop-table) - [How to Rename a Table](https://docs.databend.com/sql/sql-commands/ddl/table/ddl-rename-table) - [How to Truncate a Table](https://docs.databend.com/sql/sql-commands/ddl/table/ddl-truncate-table) - [How to Flash Back a Table](https://docs.databend.com/sql/sql-commands/ddl/table/flashback-table) - [How to Add/Drop Table Column](https://docs.databend.com/sql/sql-commands/ddl/table/alter-table-column)

Managing Data

- [COPY-INTO](https://docs.databend.com/sql/sql-commands/dml/dml-copy-into-table) - [INSERT](https://docs.databend.com/sql/sql-commands/dml/dml-insert) - [DELETE](https://docs.databend.com/sql/sql-commands/dml/dml-delete-from) - [UPDATE](https://docs.databend.com/sql/sql-commands/dml/dml-update) - [REPLACE](https://docs.databend.com/sql/sql-commands/dml/dml-replace) - [MERGE-INTO](https://docs.databend.com/sql/sql-commands/dml/dml-merge)

Managing Views

- [How to Create a View](https://docs.databend.com/sql/sql-commands/ddl/view/ddl-create-view) - [How to Drop a View](https://docs.databend.com/sql/sql-commands/ddl/view/ddl-drop-view) - [How to Alter a View](https://docs.databend.com/sql/sql-commands/ddl/view/ddl-alter-view)

AI Functions

- [Generating SQL with AI](https://docs.databend.com/sql/sql-functions/ai-functions/ai-to-sql) - [Creating Embedding Vectors](https://docs.databend.com/sql/sql-functions/ai-functions/ai-embedding-vector) - [Computing Text Similarities](https://docs.databend.com/sql/sql-functions/ai-functions/ai-cosine-distance) - [Text Completion with AI](https://docs.databend.com/sql/sql-functions/ai-functions/ai-text-completion)

Data Management

- [Data Lifecycle in Databend](https://docs.databend.com/guides/data-management/data-lifecycle) - [Data Recovery in Databend](https://docs.databend.com/guides/data-management/data-recovery) - [Data Protection in Databend](https://docs.databend.com/guides/data-management/data-protection) - [Data Purge in Databend](https://docs.databend.com/guides/data-management/data-recycle)

Accessing Data Lake

- [Apache Hive](https://docs.databend.com/guides/access-data-lake/hive) - [Apache Iceberg](https://docs.databend.com/guides/access-data-lake/iceberg/iceberg-engine) - [Delta Lake](https://docs.databend.com/guides/access-data-lake/delta)

Security

- [Access Control](https://docs.databend.com/guides/security/access-control) - [Masking Policy](https://docs.databend.com/guides/security/masking-policy) - [Network Policy](https://docs.databend.com/guides/security/network-policy) - [Password Policy](https://docs.databend.com/guides/security/password-policy)

Performance

- [Review Clickbench](https://databend.com/blog/clickbench-databend-top) - [TPC-H Benchmark: Databend Cloud vs. Snowflake](https://docs.databend.com/guides/benchmark/tpch) - [Databend vs. Snowflake: Data Ingestion Benchmark](https://docs.databend.com/guides/benchmark/data-ingest)

🤝 Contributing

Databend thrives on community contributions! Whether it's through ideas, code, or documentation, every effort helps in enhancing our project. As a token of our appreciation, once your code is merged, your name will be eternally preserved in the system.contributors table.

Here are some resources to help you get started:

👥 Community

For guidance on using Databend, we recommend starting with the official documentation. If you need further assistance, explore the following community channels:

Slack (For live discussion with the Community)
GitHub (Feature/Bug reports, Contributions)
Twitter (Get the news fast)
I'm feeling lucky (Pick up a good first issue now!)

🛣️ Roadmap

Stay updated with Databend's development journey. Here are our roadmap milestones:

📜 License

Databend is released under a combination of two licenses: the Apache License 2.0 and the Elastic License 2.0.

When contributing to Databend, you can find the relevant license header in each file.

For more information, see the LICENSE file and Licensing FAQs.

🙏 Acknowledgement

Inspiration: Databend's design draws inspiration from industry leaders ClickHouse and Snowflake.
Computing Model: Our computing foundation is built upon apache arrow.
Documentation Hosting: The Databend documentation website proudly runs on Vercel.

databendlabs / databend

readme