Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats. Working with compute engines including Flink, Spark, and Trino, Amoro brings pluggable and self-managed features for Lakehouse to provide out-of-the-box data warehouse experience, and helps data platforms or products easily build infra-decoupled, stream-and-batch-fused and lake-native architecture.
Here is the architecture diagram of Amoro:
Amoro can manage tables of different table formats, similar to how MySQL/ClickHouse can choose different storage engines. Amoro meets diverse user needs by using different table formats. Currently, Amoro supports four table formats:
Iceberg format tables use the engine integration method provided by the Iceberg community. For details, please refer to: Iceberg Docs.
Amoro support multiple processing engines for Mixed format as below:
Processing Engine | Version | Batch Read | Batch Write | Batch Overwrite | Streaming Read | Streaming Write | Create Table | Alter Table |
---|---|---|---|---|---|---|---|---|
Flink | 1.15.x, 1.16.x, 1.17.x | ✔ | ✔ | ✖ | ✔ | ✔ | ✔ | ✖ |
Spark | 3.1, 3.2, 3.3 | ✔ | ✔ | ✔ | ✖ | ✖ | ✔ | ✔ |
Hive | 2.x, 3.x | ✔ | ✖ | ✔ | ✖ | ✖ | ✖ | ✔ |
Trino | 406 | ✔ | ✖ | ✔ | ✖ | ✖ | ✖ | ✔ |
Amoro contains modules as below:
amoro-core
contains core abstractions and common implementation for other modulesamoro-ams
is amoro management service module
amoro-ams-api
contains ams thrift api and common interfacesamoro-ams-dashboard
is the dashboard frontend for amsamoro-ams-server
is the backend server for amsamoro-ams-optimizer
provides default optimizer implementationamoro-mixed-format
provides Mixed format implementation
amoro-mixed-format-hive
integrates with Apache Hive and implements Mixed Hive formatamoro-mixed-format-flink
provides Flink connectors for Mixed format tables (use amoro-flink-runtime for a shaded version)amoro-mixed-format-spark
provides Spark connectors for Mixed format tables (use amoro-spark-runtime for a shaded version)amoro-mixed-format-trino
provides Trino connectors for Mixed format tablesAmoro is built using Maven with JDK 8 and JDK 17(only for amoro-mixed-format/amoro-mixed-format-trino
module).
amoro-mixed-format-trino
: mvn clean package
mvn clean package -DskipTests
mvn clean package -Pskip-dashboard-build
mvn clean package -DskipTests -Phadoop2
mvn clean package -DskipTests -Dflink-optimizer.flink-version=1.15.4
-Pflink-optimizer-pre-1.15
parameter: mvn clean package -DskipTests -Pflink-optimizer-pre-1.15 -Dflink-optimizer.flink-version=1.14.6
mvn clean package -DskipTests -Dspark-optimizer.spark-version=3.2.2
amoro-mixed-format-trino
module under JDK 17: mvn clean package -DskipTests -Ptrino-spotless,build-mixed-format-trino -pl 'amoro-mixed-format/amoro-mixed-format-trino' -am
.mvn clean package -DskipTests -Ptoolchain,build-mixed-format-trino
, besides you need config toolchains.xml
in ${user.home}/.m2/
dir with content below.<?xml version="1.0" encoding="UTF-8"?>
<toolchains>
<toolchain>
<type>jdk</type>
<provides>
<version>17</version>
<vendor>sun</vendor>
</provides>
<configuration>
<jdkHome>${YourJDK17Home}</jdkHome>
</configuration>
</toolchain>
</toolchains>
Visit https://amoro.apache.org/quick-demo/ to quickly explore what amoro can do.
If you are interested in Lakehouse, Data Lake Format, welcome to join our community, we welcome any organizations, teams and individuals to grow together, and sincerely hope to help users better use Data Lake Format through open source.
Join the Amoro WeChat Group: Add " kllnn999
" as a friend on WeChat and specify "Amoro lover".
This project exists thanks to all the people who contribute.
Made with contrib.rocks.