search

alexreidy / parallafka-for-dotnet

Parallelized Kafka consumer for maximizing throughput
Apache License 2.0
1 stars 1 forks source link

adds GetStats to return anonymous object of statistics #7

Closed Gonyoda closed 2 years ago

Gonyoda commented 2 years ago

You can get stats that when serialized over JSON, look like this:

{
  "ConsumerState": "Polling for Kafka messages",
  "MaxQueuedMessages": 1000,
  "CallDuration": "0:00:05.3857492",
  "CallDurationMs": 5385,
  "CommitState": [
    {
      "Partition": 1,
      "CommitsPending": 0
    },
    {
      "Partition": 0,
      "CommitsPending": 18
    }
  ],
  "MessagesByKey": {
    "MessageCountsByKey": {
      "21568756": 0,
      "21568776": 0,
      "21568780": 0,
      "21568781": 0,
      "21568785": 0,
      "21568787": 0,
      "21568793": 0,
      "21568794": 0
    }
  },
  "Router": {
    "MessagesRouted": 2287,
    "MessagesHandled": 2286,
    "MessagesNotHandled": 0,
    "MessagesSkipped": 0,
    "IncomingQueueSize": 0
  },
  "RoutingTarget": {
    "InputCount": 0,
    "TaskStatus": "WaitingForActivation"
  },
  "FinishedRouter": {
    "InputCount": 0,
    "MessagesHandled": 2278,
    "MessagesSent": 0,
    "MessagesNotSent": 0,
    "MessagesSkipped": 2278
  },
  "Handler": {
    "InputCount": 0,
    "MessagesHandled": 2286,
    "MessagesSent": 2278,
    "MessagesNotSent": 0,
    "MessagesErrored": 0
  },
  "HandlerTarget": {
    "InputCount": 0,
    "TaskStatus": "WaitingForActivation"
  },
  "Committer": {
    "InputCount": 1,
    "MessageCommitLoopInProgress": true,
    "MessagesCommitted": 87,
    "MessagesCommitErrors": 0,
    "MessagesCommitLoops": 134
  },
  "CommitterTarget": {
    "InputCount": 0,
    "TaskStatus": "WaitingForActivation"
  },
  "MessageHandledTarget": {
    "InputCount": 0,
    "TaskStatus": "WaitingForActivation"
  }
}
Gonyoda commented 2 years ago

Using Event Counters or Metrics.Meter is the idiomatic way of doing counters/stats in .NET Core. But accessing these counters on an as-needed basis instead of every X seconds is virtually impossible according to EventListener docs. Same for Metrics which seem to record every time the metric is changed. The approach in this PR, using a method that is called on demand is better for the current use case of these Parallafka metrics - to be displayed only on demand, not on a periodic timer.

alexreidy commented 2 years ago

Closing as it's now part of https://github.com/alexreidy/parallafka-for-dotnet/pull/8