[Request] Number of parked messages for a subscription

[ben-leah-cko]

Firstly, thanks for your work on this exporter, it's been a great help.

One thing that would be useful, is to get a metric of the number of messages in the parked queue for a subscription.

It does not look like that metric is included with the extra statistics on a subscripton but I believe it could be calculated by using the parked stream metadata to get the "truncate before" value ($tb) and subtracting that from the latest event number in the parked stream.

/streams/$persistentsubscription-$ce-CategoryStream::Subscription-Group-parked/metadata

{
  "$tb": 1185,
  "$acl": {
     }
}

[marcinbudny]

Hi Ben,

Thank you for the suggestion. I'm not very familiar with persistent subscriptions, because I don't use them myself. If I understand this correctly, for each subscription group the exporter would have to:

1. call the metadata endpoint
2. read last event from the stream

I'm a bit concerned about the performance of this operation when there is a lot of groups. Scrape needs to complete within a few seconds, preferably under a second. So maybe there should be an opt-in setting for the exporter to scrape such metric. What do you think?

[ben-leah-cko]

Hey Marc,

Yeah, I completely understand the performance angle. Possible option

Potentially make the opt-in on a per subscription (group) basis. As the number of parked messages (it's essentially a dead letter queue), is generally quite slow-changing, you could config the scrape to happen at longer intervals?

[marcinbudny]

The stats are downloaded from ES when the scrape is triggered by Prometheus. This is how exporters are supposed to work, so cadence will be the same for all the data. I'm not sure about per subscription group configuration, it may be a bit too much. I could try with simple on/off switch for this and then fix this later if needed. That being said, I'm a bit overwhelmed by work right now, so I won't be able to start this for few next weeks. If you need this sooner and you would like to contribute a PR, please feel free to do so.

[alexeyzimarev]

@marcinbudny that's not exactly right. The persistent subscriptions HTTP API returns everything you need to report as a metric. There is a single endpoint to list all the persistent subscriptions: https://ecstatic-borg-bc8046.netlify.app/server/5.0/http-api/persistent-subscriptions.html#acknowledgements

[alexeyzimarev]

The endpoint also returns the URI for parked messages, I haven't explored it myself, but the Admin UI is showing the number of parked and failed messages, so it should not be hard.

[marcinbudny]

Hi Alexey. I'm already using /subscriptions to get some information about persistent subscriptions, but number of parked messages requires more advanced approach. I need to look into this further.

[alexeyzimarev]

Let me know if something isn't clear, I can reach out people in dev.

[marcinbudny]

Resolved in #14