grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.97k stars 3.46k forks source link

refactor(kafka-reader): Separates kafka constructs for better separation of concerns & reuse. #14982

Closed owen-d closed 9 hours ago

owen-d commented 6 days ago

Refactor Kafka Reader and Partition Committer

This PR refactors the Kafka ingestion pipeline components to achieve better separation of concerns and cleaner interfaces. The changes split the existing Reader into three distinct components:

  1. A ReaderIfc that handles pure Kafka interactions
  2. A partitionCommitter that manages async offset commits (already existed, but I've refactored it to use the interface)
  3. A ReaderService that coordinates the overall lifecycle

Key Changes

Note: As of now, this just adds 3 new files with refactored versions -- if we're happy I'll replace the existing ones, but the current structure mirrors how I developed against the prior versions.

~Disclaimer: I haven't hooked this up to testware yet; if we're happy with this direction, I'll do that next.~ edit: done.

New Reader Interface

Created a focused interface for Kafka operations:

type ReaderIfc interface {
    Topic() string
    Partition() int32
    ConsumerGroup() string
    FetchLastCommittedOffset(ctx context.Context) (int64, error)
    FetchPartitionOffset(ctx context.Context, position SpecialOffset) (int64, error)
    Poll(ctx context.Context) ([]Record, error)
    Commit(ctx context.Context, offset int64) error
    // Set the target offset for consumption. reads will begin from here.
    SetOffsetForConsumption(offset int64)
}

Improved Committer Design

Service Lifecycle

Benefits