AbsaOSS / hyperdrive

Extensible streaming ingestion pipeline on top of Apache Spark
Apache License 2.0
44 stars 13 forks source link

OffsetManager should not expect topic as argument #87

Closed kevinwallimann closed 4 years ago

kevinwallimann commented 4 years ago

Description Currently, the base class OffsetManager expects a topic as a constructor argument. This is too specific to the use case of the CheckpointOffsetManager which currently requires the kafka topic (will be resolved in #85 ). In fact, this component does not need to manage offsets, but may manage anything which concerns both DataStreamReader and DataStreamWriter.

Tasks

How to migrate