Stargazers Over Time | Contributors Over Time |
---|---|
Apache InLong is a one-stop, full-scenario integration framework for massive data that supports Data Ingestion
, Data Synchronization
and Data Subscription
, and it provides automatic, secure and reliable data transmission capabilities. InLong also supports both batch and stream data processing at the same time, which offers great power to build data analysis, modeling and other real-time applications based on streaming data.
InLong (应龙) is a divine beast in Chinese mythology who guides the river into the sea, and it is regarded as a metaphor of the InLong system for reporting data streams.
InLong was originally built at Tencent, which has served online businesses for more than 8 years, to support massive data (data scale of more than 80 trillion pieces of data per day) reporting services in big data scenarios. The entire platform has integrated 5 modules: Ingestion, Convergence, Caching, Sorting, and Management, so that the business only needs to provide data sources, data service quality, data landing clusters and data landing formats, that is, the data can be continuously pushed from the source to the target cluster, which greatly meets the data reporting service requirements in the business big data scenario.
For getting more information, please visit our project documentation at https://inlong.apache.org/.
Apache InLong offers a variety of features:
InLong aims to provide a one-stop, full-scenario integration framework for massive data, users can easily build stream-based data applications. It supports Data Ingestion
, Data Synchronization
and Data Subscription
at the same time, and is suitable for environments that need to quickly build a data reporting platform, as well as an ultra-large-scale data reporting environment that InLong is very suitable for, and an environment that needs to automatically sort and land the reported data.
You can use InLong in the following ways:
Type | Name | Version |
---|---|---|
Extract Node | Auto Push | None |
File | None | |
Kafka | 2.x | |
MongoDB | >= 3.6 | |
MQTT | >= 3.1 | |
MySQL | 5.6, 5.7, 8.0.x | |
Oracle | 11,12,19 | |
PostgreSQL | 9.6, 10, 11, 12 | |
Pulsar | 2.8.x | |
Redis | 2.6.x | |
SQLServer | 2012, 2014, 2016, 2017, 2019 | |
Load Node | Auto Consumption | None |
ClickHouse | 20.7+ | |
Elasticsearch | 6.x, 7.x | |
Greenplum | 4.x, 5.x, 6.x | |
HBase | 2.2.x | |
HDFS | 2.x, 3.x | |
Hive | 1.x, 2.x, 3.x | |
Iceberg | 0.12.x | |
Hudi | 0.12.x | |
Kafka | 2.x | |
MySQL | 5.6, 5.7, 8.0.x | |
Oracle | 11, 12, 19 | |
PostgreSQL | 9.6, 10, 11, 12 | |
SQLServer | 2012, 2014, 2016, 2017, 2019 | |
TDSQL-PostgreSQL | 10.17 | |
Doris | >= 0.13 | |
StarRocks | >= 2.0 | |
Kudu | >= 1.12.0 | |
Redis | >= 3.0 |
More detailed instructions can be found at Quick Start section in the documentation.
Requirements:
CodeStyle:
mvn spotless:apply
Compile and install:
mvn clean install -DskipTests
(Optional) Compile using docker image:
docker pull maven:3.6-openjdk-8
docker run -v `pwd`:/inlong -w /inlong maven:3.6-openjdk-8 mvn clean install -DskipTests
after compile successfully, you could find distribution file at inlong-distribution/target
.
Join Apache InLong mailing lists: | Name | Scope | |||
---|---|---|---|---|---|
dev@inlong.apache.org | Development-related discussions | Subscribe | Unsubscribe | Archives |
© Contributors Licensed under an Apache-2.0 license.