Question about metrics-datadog usage

[s101d1]

I have Dropwizard app with a Resource class and its method annotated with @Timed annotation:

package com.example.resources;

@Path("/users")
public class UserResource {

    @GET
    @Path("/{id}")
    @Timed
    public User findUser(@PathParam("id") LongParam id) {
        ...
    }

}

When I opened http://localhost:8081/metrics?pretty=true on browser, I can see the findUser method metrics data, such as:

"timers" : {
    "com.example.resources.UserResource.findUser" : {
        "count" : 1,
        "max" : 0.042068400000000006,
        "mean" : 0.042068400000000006,
        "min" : 0.042068400000000006,
        "p50" : 0.042068400000000006,
        "p75" : 0.042068400000000006,
        "p95" : 0.042068400000000006,
        "p98" : 0.042068400000000006,
        "p99" : 0.042068400000000006,
        "p999" : 0.042068400000000006,
        "stddev" : 0.0,
        "m15_rate" : 0.0,
        "m1_rate" : 0.0,
        "m5_rate" : 0.0,
        "mean_rate" : 0.026105587280777456,
        "duration_units" : "seconds",
        "rate_units" : "calls/second"
    },
    ...
}

But I don't see the same metric data in DataDog Agent Manager logs.

Most of metrics data I see are jvm related metrics such as:

(as showed in "Logs & Status" -> "Dogstatsd Logs")

...
jvm.memory.pools.Metaspace.init:0|g
jvm.memory.pools.Metaspace.max:-1|g
jvm.memory.pools.Metaspace.usage:0.977711|g
jvm.memory.pools.Metaspace.used:39686600|g`
Traceback (most recent call last):
  File "dogstatsd.pyc", line 420, in start
  File "aggregator.pyc", line 612, in submit_packets
  File "aggregator.pyc", line 485, in parse_metric_packet
Exception: Metric value must be a number: jvm.filedescriptor, ?

FYI this is the code where I enable the DataDog metrics:

public class MainApplication extends Application<MainConfiguration> {

    @Override
    public void run(final MainConfiguration configuration, final Environment environment) throws IOException {
        EnumSet<DatadogReporter.Expansion> expansions = DatadogReporter.Expansion.ALL;
        UdpTransport udpTransport = new UdpTransport.Builder().build();
        DatadogReporter reporter = DatadogReporter.forRegistry(environment.metrics())
            .withTransport(udpTransport)
            .withExpansions(expansions)
            .build();

        reporter.start(10, TimeUnit.SECONDS);
    }
}

and in the dropwizard yml file:

metrics:
  frequency: 1 minute
  reporters:
    - type: datadog
      transport:
        type: udp

I'm new to dropwizard metrics so perhaps I missed something here?

Also, what is the difference between frequency option in yml and period time in reporter.start(10, TimeUnit.SECONDS) ?

[nickdella]

Hi @s101d1 ! Unfortunately, I'm not very familiar with Dropwizard either; we use Play Framework here at Coursera. I dug around Dropwizard a bit and it looks like you shouldn't need to start the reporter on your own. Dropwizard should pick up your configuration from the yaml file and start DatadogReporterautomatically (see https://github.com/dropwizard/dropwizard/blob/7295c5faf42840325c161cc82ad4ed1db2bf8419/dropwizard-metrics/src/main/java/io/dropwizard/metrics/ScheduledReporterManager.java#L32 ). Can you try removing the reporter initialization from your run method?

[s101d1]

I removed them and still don't see metric data of "findUser" in DataDog Agent logs.

[s101d1]

If I add this settings on config.yml, it shows the findUser metrics info on the console:

metrics:
  frequency: 1 minute
  reporters:
    - type: datadog
      transport:
        type: udp
    - type: console
          output: stdout

Not sure if the DataDog Agent Manager app has bug or I'm missing someting here.

[aantono]

Make sure you have a DataDog agent and a DogStatsD listener running on the localhost where your application is running. The reporter under the hood sends data to DogStatsD endpoint, which, by default, is expected to be running on localhost:8125.