not able to see to custom metric(micrometer) published on dynatrace (log enabled)

[Krithikavv]

Describe the bug im working on integrating micrometer with dynatrace for monitoring purpose. I try to run spring boot app with spring boot actuator /config specific for Dynatrace integration on micrometer docs page. used the same configs. Tried changing log level but no luck to see custom metric. only the system metrics are exported from micrometer.

I see this , but metrics are not getting exported from app still.

[Loaded com.dynatrace.metric.util.DynatraceMetricApiConstants from file:/home/anscdev2/AN.gANHanaTeam-2355/spring-boot-jars/lib/com.dynatrace.metric.util.dynatrace-metric-utils-java-1.1.0.jar]
2022-03-09 22:29:50.459  INFO 23143 --- [           main] i.m.dynatrace.v2.DynatraceExporterV2     : Exporting to endpoint https://apm.cf.us30.hana.ondemand.com/e/e584d4c6-69cf-459a-a685-2ad0673f5ff7/api/v2/metrics/ingest
[Loaded com.dynatrace.metric.util.MetricBuilderFactory from file:/home/anscdev2/AN.gANHanaTeam-2355/spring-boot-jars/lib/com.dynatrace.metric.util.dynatrace-metric-utils-java-1.1.0.jar]
[Loaded com.dynatrace.metric.util.MetricBuilderFactory$MetricBuilderFactoryBuilder from file:/home/anscdev2/AN.gANHanaTeam-2355/spring-boot-jars/lib/com.dynatrace.metric.util.dynatrace-metric-utils-java-1.1.0.jar]
[Loaded java.util.Collections$2 from /usr/j2sdk1.8.0_302_x86_64-an-tzdata2021a-ca/jre/lib/rt.jar]

config on spring boot app: not able to see any logs or details from micrometer dyantrace . also publishing is not happening to dynatrace. searched for spring.integration. as per standard spring docs support.

spring:
  profiles: default
  servlet:
    multipart:
      max-file-size: 200MB
      max-request-size: 205MB
server:
#application context root url path, this should match webserver config which is derived from
#Tomcat.ApplicationContext from DD.xml + appflags.cfg ServletNames, this is default, can be
#overridden in application.properties
  servlet:
    contextPath: path
micro:
  server.contextPath: path
an:
  appName: app1
  loadANPermissions: true
#  jpa.eclipselink.logging.level: FINE
#  jpa.eclipselink.logging.level.sql: FINE
db:
  modelGroupName: scmMicro
queue:
  servers: localhost:9123
  consumers:
  ---
spring:
  profiles: dev
server:
  servlet:
    contextPath:
management:
  server:
    port:

---
management:
  endpoints:
    metrics:
      enabled:
        - true
    web:
      exposure:
        include:
          - health
          - metrics
          - dynatrace

---
logging.level.io.micrometer.dynatrace: DEBUG

---
management.metrics.export.dynatrace:
   # Required only if not using the OneAgent endpoint
   # For SaaS: https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest
   # For managed deployments: https://{your-domain}/e/{your-environment-id}/api/v2/metrics/ingest
   uri: <https://uri>/api/v2/metrics/ingest

   # should be read from a secure source
   api-token: XXXX

   # These properties can only be used with the v2 exporter.
   v2:
     # Sets a prefix that is prepended to each exported metric key.
     metric-key-prefix: dynatest2

     # If set to true and a local OneAgent or operator is running, retrieves metadata
     # and adds it as additional dimensions to all data points (default: true)
     enrich-with-dynatrace-metadata: true

     # Sets an arbitrary number of key-value pairs as default dimensions.
     # Micrometer tags will overwrite these dimensions, if they have the same key.
     # Each exported metric will contain these dimensions.
     default-dimensions:
       stack: "dev"

     # The export interval in which metrics are sent to Dynatrace (default: 60s).
     step: 30s

Environment from server hosted : managed host machine. https://apm.cf.us30.hana.ondemand.com/e/e584d4c6-69cf-459a-a685-2ad0673f5ff7/ui/metrics?gtf=-2h&gf=all latest version : 1.8.3 (micrometer) dyna micrometer registry : 1.8.3 OS: linux java: 1.8

java version "1.8.0_301" Java(TM) SE Runtime Environment (build 1.8.0_301-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.301-b09, mixed mode) To Reproduce How to reproduce the bug: expose any functionality API call from spring boot app to dynatrace with any metrics (timer,counter etc)

Expected behavior metrics should flow into dynatrace with name spring.integration. as name

Additional context Add any other context about the problem here, e.g. related issues.

[shakuzen]

@Krithikavv I'm not sure you want to be sharing your API token here as this is public and viewable by anyone. I have edited the issue description to redact it, but anyone watching this repository has potentially seen it. You probably should delete that API token and generate a new one to use so others don't use yours.

@pirgeo I'm not sure if you're able to help here, or should @Krithikavv open a support ticket through Dynatrace's support channels instead?

[pirgeo]

@shakuzen thanks for redacting it. I agree that you should delete it and create a new one to make sure nobody else can use it.

@Krithikavv Do I understand correctly, that system metrics are correctly exported and show up in Dynatrace, but custom metrics that you created yourself in code do not show up? If that is the case, @shakuzen, do you know if there is something one needs to specify explicitly to export custom metrics from Spring Boot as well? My understanding was that all metrics collected (including custom ones) will automatically be exported to all registries.

There is already an issue on the Dynatrace Community forums. If this issue persists, please consider opening a support ticket, so our team can take a closer look. Thanks!

[Krithikavv]

Hi sure, thanks yes I removed the token and uri and details and changed it. outside world wont have access to the secure page as well. thanks for heads up. yes I was also thinking that spring boot apps must bring out all the metrics including custom metrics. I have called custom metrics in my app and able to see the details. Between, yes I have used simple meter registry to export contents to dynatrace and spring boot apps have config to talk to dynatrace. I don't see this custom metrics (say counter, timer,etc) published to exporter. should I use dynatracemeterregistry ? for this production integration instead of simple meter registry.

[Krithikavv]

Hi guys, I guess I was looking at simple meter registry so was not able to see details in prod. changed to dynatrace meter registry and able to see them in dynatrace. but my qq is some of the meter values are empty and not proper in this dynatrace meter registry which I saw in simple meter registry.. let me just put that alone here.. I don't see mean/count of event(which was one in simple meter registry ) /total time.. why does this discrepancy arise ?

.a.m.c.micrometertest.MicroLibImpl : event count metric:0 2022-03-11 12:41:05.411 INFO 12651 --- [nio-9401-exec-9] c.a.m.c.micrometertest.MicroLibImpl : events totaltime ns:0.0 2022-03-11 12:41:05.411 INFO 12651 --- [nio-9401-exec-9] c.a.m.c.micrometertest.MicroLibImpl : single event maxtime ns:118.121266 2022-03-11 12:41:05.411 INFO 12651 --- [nio-9401-exec-9] c.a.m.c.micrometertest.MicroLibImpl : print all events mean ns:0.0

[pirgeo]

Hi @Krithikavv, Thanks for reporting back that using the DynatraceMeterRegistry fixed the problem of metrics not showing up in Dynatrace.

As for your second question: Does the problem only show up when working with Timers in nanosecond resolution, or is this the case for other instruments too (especially for non-time related ones, like DistributionSummary)?

[Krithikavv]

hi so basically I see the metric not printing for even counter(increment still showed 0) or distribution summary type of objects as well not just timer.

[Krithikavv]

Hi do I need to raise a ticket for this separately ? I don't see most of details printed.. even for counter increment() is not wokring to export detail to dynatrace.

[pirgeo]

For looking further into this problem it would be helpful to have access to your environment and this would have to by handled by our support team, so if you'd like to get their assistance please open a support ticket with them.

[Krithikavv]

Hi sure, is there a uri that you can guide me so that I can raise a support ticket ?

[pirgeo]

Hi, please take a look here: https://support.dynatrace.com/. There should be a link for you to open a support ticket there.

[shakuzen]

It sounds like the originally reported problem has been solved. If there's any change needed in Micrometer related to the new issue, let's track that in a new issue with details once support has had a chance to look at it. Thanks for the help @pirgeo.

In case it helps in looking into this, SimpleMeterRegistry by default reports cumulative values whereas DynatraceMeterRegistry is a StepMeterRegistry, which means it reports a value per step for the previous step interval. For example, if you increment a counter, the increment is reflected in the count during the next step interval. This is described in the documentation under Rate Aggregation