micrometer-metrics / micrometer

An application observability facade for the most popular observability tools. Think SLF4J, but for observability.
https://micrometer.io
Apache License 2.0
4.45k stars 980 forks source link

Distribution value has |bucket_counts| fields that sum to 4 which does not equal the |count| field value of 2 #4868

Open altunkan opened 6 months ago

altunkan commented 6 months ago

Describe the bug Hello,

Recently we have migrated our Spring Boot application from 2.7 to 3.2 and we've started to observe following error.

com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: Field timeSeries[31].points[0].distributionValue had an invalid value: Distribution value has |bucket_counts| fields that sum to 4 which does not equal the |count| field value of 2.

Environment

To Reproduce How to reproduce the bug:

  private Timer buildDependencyTimer(DependencyTimer dependencyTimer) {
    return Timer.builder(LATENCY_DEPENDENCY_PREFIX)
        .tags(getDependencyTags(dependencyTimer))
        .publishPercentiles(percentiles)
        .publishPercentileHistogram()
        .minimumExpectedValue(Duration.ofNanos(1L))
        .register(meterRegistry);
  }

    dependencyLatencyTimers.putIfAbsent(dependencyTimer, buildDependencyTimer(dependencyTimer));
    dependencyLatencyTimers.get(dependencyTimer).record(responseTime, TimeUnit.MILLISECONDS);

Expected behavior Metrics should be reported to stackdriver.

Additional context Add any other context about the problem here, e.g. related issues.

hudson155 commented 5 months ago

Seeing the same thing on a new project @altunkan is this new after a version upgrade? I might try older versions...

hudson155 commented 5 months ago

I got it working by downgrading to 1.11.6.

altunkan commented 5 months ago

Yeah, I just tried with 1.11.6 by adding set('micrometer.version', '1.11.6') and looks like error is gone. Thank you! This will keep us good until micrometer team addresses the issue.

rafal-dudek commented 4 months ago

I believe this is the same issue as that: https://github.com/micrometer-metrics/micrometer/issues/998 The problem is there since 2018 but looks like on Spring Boot 3.2 it is much more common. We had a sporadic problem with it in our company since 2021, but after migration to Spring Boot 3.2 we have already dozens of applications experiencing it regularly.

altunkan commented 3 months ago

Unfortunately bug still happens with version 1.11.6.

All of our count based metrics fluctuate and we disabled error logging for that specific class.