dconnett-splunk / splunk-add-on-for-dynatrace

Apache License 2.0
0 stars 0 forks source link

Too much metrics with the same timestamp #4

Closed pdros-splunk closed 7 months ago

pdros-splunk commented 10 months ago

Some of the customer report that after some time add-on stop to add metrics in Splunk, this can be due to the same timestamp for a lot of metrics, according to this message from customer:

From Customer: I think the issue will center around this error in internal logs. 01-22-2024 06:54:02.724 -0600 WARN AggregatorMiningProcessor [5267 merging] - Too many events (100K) with the same timestamp: incrementing timestamps 1 second(s) into the future to insure retrievability - data_source="dynatrace_timeseries_metrics://dynatrace_time_series_metrics", data_host="splk.customer.com", data_sourcetype="dynatrace:metrics" 01-22-2024 06:33:21.543 -0600 WARN DateParserVerbose [5267 merging] - The same timestamp has been used for 100K consecutive times. If more than 200K events have the same timestamp, not all events may be retrieveable. Context: source=dynatrace_timeseries_metrics://dynatrace_time_series_metrics|host=splk.customer.com.com|dynatrace:metrics|\n 1 similar messages suppressed. First occurred at: Mon Jan 22 06:24:01 2024

pdros-splunk commented 10 months ago

Adding idea to add one millisecond to every timestamp as time event, to avoid duplicates in the time of events

dconnett-splunk commented 7 months ago

Fixed in following commit: https://github.com/dconnett-splunk/splunk-add-on-for-dynatrace/commit/8ab98bb8f2b298312688f4450febec242884bda0