Kappa Architecture

[ranjeet-floyd]

Description: The Kappa Architecture is a data processing architecture that provides a simplified approach to handling both real-time and batch data processing. Unlike the Lambda Architecture, which requires separate paths for batch and real-time processing, the Kappa Architecture uses a single stream processing engine for both real-time and historical data processing.

Main elements of the Kappa Design Pattern include:

• Single Data Pipeline: A unified data processing stream that handles both real-time and historical data.
• Stream Processing Engine: A core component that processes incoming data in real-time.
• Immutable Data Store: All data is stored in an immutable, append-only log, ensuring data integrity and enabling easy reprocessing.
• Reprocessing Capability: The ability to reprocess historical data by replaying the log.

References:

1. Kappa Architecture by Milinda Pathirage
2. Kappa Architecture GitHub Repository
3. Java Design Patterns Contribution Guidelines

Acceptance Criteria:

1. Implement a single data pipeline that processes both real-time and historical data.
2. Integrate a stream processing engine to handle incoming data in real-time.
3. Ensure that the data store is immutable and supports reprocessing of historical data by replaying the log.

[ranjeet-floyd]

@iluwatar Can i take this ?. Thanks

[iluwatar]

Go ahead @ranjeet-floyd

[iluwatar]

The issue is free to take

[Azureyjt]

@iluwatar May I work on this issue?

[iluwatar]

Ok @Azureyjt

[Azureyjt]

Ok @Azureyjt

Thanks iluwatar. And I'm afraid that I need to study the background first since I'm not very familiar with big data system and streaming process. So it may take a little bit more time.

[iluwatar]

No problem. Thanks for looking into this.

[Anurag870]

@iluwatar I would like to take it up. Please assign this to me.

[iluwatar]

Great @Anurag870, it's done

[Anurag870]

@iluwatar Kappa Architecture requires log-based storage such as Kafka or Pulsar and a stream processing engine such as Spark or Flink. Having a basic running Java example would mean setting up these systems as well.

Even writing UML diagrams for it may not be very suitable.

I was thinking if I can add draw.io image and xml for this along with some description. If you feel it is off our scope, we can close the issue as well.

[iluwatar]

The approach we used in Hexagonal Architecture was to have multiple implementations of data store. There was an in-memory data store and then with some additional configuration you could switch to Mongo database.

[Anurag870]

Sure will look at that

[iluwatar]

@Anurag870 still working on this?

[iluwatar]

The issue is unassigned again