Consolidate metrics under the sidekiq.* namespace

[weppos]

Ruby version: - Rails version: - Sidekiq / Pro / Enterprise version(s): 7.x

Sidekiq Pro sends metrics under the following namespaces:

sidekiq.*
jobs.*
batches.*
leader.election

This behavior is undesirable as, when blended into a large organization account, you can easily miss metrics. There are options to namespace the metrics, but the namespace is applied to all metrics.

So for instance, if I set the "sidekiq" namespace, as it follows:

config.dogstatsd = -> { Datadog::Statsd.new("1.2.3.4", "8125", namespace: "sidekiq") }

I will obtain the following:

sidekiq.sidekiq.*
sidekiq.jobs.*
sidekiq.batches.*
sidekiq.leader.election

Ideally, all metrics are reported under a consistent "sidekiq" prefix. Adjusting the namespace will pre-pend the prefix.

After some internal configuration, we obtained something similar applying a namespace, and remapping "sidekiq.sidekiq." block to "sidekiq.".

[javierjulio]

@weppos I'm dealing with the same situation setting up metrics and I was curious what you are using currently for the namespace: keyword argument in the Datadog::Statsd instance? Did you end up using namespace: "sidekiq.my-app" to get things working?

[javierjulio]

Worth noting that in the Datadog documentation for updating the Agent configuration file it does suggest matching against a "sidekiq.sidekiq." style prefix so that may have to be changed.

# dogstatsd_mapper_cache_size: 1000  # default to 1000
dogstatsd_mapper_profiles:
  - name: sidekiq
    prefix: "sidekiq."
    mappings:
      - match: 'sidekiq\.sidekiq\.(.*)'
        match_type: "regex"
        name: "sidekiq.$1"
      - match: 'sidekiq\.jobs\.(.*)\.perform'
        name: "sidekiq.jobs.perform"
        match_type: "regex"
        tags:
          worker: "$1"
      - match: 'sidekiq\.jobs\.(.*)\.(count|success|failure)'
        name: "sidekiq.jobs.worker.$2"
        match_type: "regex"
        tags:
          worker: "$1"

[weppos]

I used the recommended setup in the Agent. I set namespace: "sidekiq" in the Datadog::Statsd instance, then used the mapper to rewrite sidekiq.sidekiq entries (line 1 of 3).

The mappers 2 and 3 are technically unnecessary, if you use most recent versions of Sidekiq. They were used to map metrics where the worker name was part of the metric name, into tags. But the use of tag was already introduced in Sidekiq codebase one or two major versions ago.

[javierjulio]

@weppos thank you. That's most helpful. A final clarification, Did you happen to only have one Sidekiq running or multiple Sidekiq's for different apps? I'm guessing the former as to why the namespace could just be namespace: "sidekiq" vs it including the app/service name?

[weppos]

@weppos thank you. That's most helpful. A final clarification, Did you happen to only have one Sidekiq running or multiple Sidekiq's for different apps? I'm guessing the former as to why the namespace could just be namespace: "sidekiq" vs it including the app/service name?

I anticipated your question here 😉 https://github.com/sidekiq/sidekiq/issues/6406#issuecomment-2318779397

TL;DR: we have only one app, but it is irrelevant. Even if we had more, we would not prefix metrics by app. Instead, we use tagging which is a better way to organize the information.

We have other services like Postgres for which we run multiple instances. Same deal, the metrics are not prefixed by the instance, instead we tag each source and we use the same metric name. This allows us to view data both aggregated or isolated per tag.

Note this setup is also the one somehow implicitly assumed by the default Dashboard that Datadog makes you available when you enable the Sidekiq integration. And this is how they structure Dashboard and metrics across most services.

[mperham]

Thanks @weppos, I forgot about tags. Tags offer different way of organizing your metrics and may lead to a different set of capabilities when viewing/querying/filtering. YMMV!