Pravega Flink Connectors

This repository implements connectors to read and write Pravega Streams with Apache Flink stream processing framework.

The connectors can be used to build end-to-end stream processing pipelines (see Samples) that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams.

Features & Highlights

Exactly-once processing guarantees for both Reader and Writer, supporting end-to-end exactly-once processing pipelines
Seamless integration with Flink's checkpoints and savepoints.
Parallel Readers and Writers supporting high throughput and low latency processing.
Table API support to access Pravega Streams for both Batch and Streaming use case.

Compatibility Matrix

The master branch will always have the most recent supported versions of Flink and Pravega.

Git Branch	Pravega Version	Flink Version	Status	Artifact Link
master	0.14	1.18	Under Development	https://github.com/pravega/flink-connectors/packages/19676441
r0.14-flink1.17	0.14	1.17	Under Development	https://github.com/pravega/flink-connectors/packages/1441637
r0.14-flink1.16	0.14	1.16	Under Development	https://github.com/pravega/flink-connectors/packages/1704300
r0.13	0.13	1.16	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.16_2.12/0.13.0/
r0.13-flink1.15	0.13	1.15	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.15_2.12/0.13.0/
r0.13-flink1.14	0.13	1.14	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.14_2.12/0.13.0/
r0.12	0.12	1.15	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.15_2.12/0.12.0/
r0.12-flink1.14	0.12	1.14	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.14_2.12/0.12.0/
r0.12-flink1.13	0.12	1.13	Released	https://repo1.maven.org/maven2/io/pravega/pravega-connectors-flink-1.13_2.12/0.12.0/

How to build

Building the connectors from the source is only necessary when we want to use or contribute to the latest (unreleased) version of the Pravega Flink connectors.

To build the project, Java version 11 is required and the repository needs to be checkout via git clone https://github.com/pravega/flink-connectors.git.

The connector project is linked to a specific version of Pravega, based on the pravegaVersion field in the gradle.properties.

After cloning the repository, the project can be built (excluding tests) by running the below command in the project root directory flink-connectors.

./gradlew clean build -x test

How to use

Check out documents here to learn how to build your own applications using Flink connector for Pravega. Also watch out that the Java version required to run the connector is either 8 or 11.

More examples on how to use the connectors with Flink application can be found in Pravega Samples repository.

Support

Don't hesitate to ask! Contact the developers and community on Slack (signup) if you need any help. Open an issue if you found a bug on Github Issues.

About

Flink connectors for Pravega is 100% open source and community-driven. All components are available under Apache 2 License on GitHub.