search

NetApp / harvest

Open-metrics endpoint for ONTAP and StorageGRID
https://netapp.github.io/harvest/latest
Apache License 2.0
150 stars 36 forks source link

ZapiPerf pollCounter error #2829

Closed rahulguptajss closed 6 months ago

rahulguptajss commented 6 months ago

ZapiPerf collector encounters a failure when executing PollCounter, resulting in less instances being exported. This issue happens from the second invocation of PollCounter and persists in subsequent calls. The regression appears to have been introduced by the changes in PR #2813

image 
024-04-12T13:59:52+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=2786 bytesRx=327205 calcMs=4 collector=ZapiPerf:WorkloadVolume exportMs=47 instances=188 instancesExported=188 metrics=5076 metricsExported=4512 numCalls=1 numPartials=0 parseMs=386 pluginMs=0 pollMs=0 skips=0 zBegin=1712910588994
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [sequential_writes] error="duplicate metric key => sequential_writes" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [read_latency] error="duplicate metric key => read_latency" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [sequential_reads] error="duplicate metric key => sequential_reads" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [write_data] error="duplicate metric key => write_data" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [write_latency] error="duplicate metric key => write_latency" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [write_ops] error="duplicate metric key => write_ops" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [read_ops] error="duplicate metric key => read_ops" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [latency] error="duplicate metric key => latency" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [other_ops] error="duplicate metric key => other_ops" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [ops] error="duplicate metric key => ops" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [total_data] error="duplicate metric key => total_data" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [concurrency] error="duplicate metric key => concurrency" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1395 > add array metric element [read_io_type.cache]:  error="duplicate metric key => read_io_type.cache" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 ERR zapiperf/zapiperf.go:1424 > add scalar metric [read_data] error="duplicate metric key => read_data" Poller=u2 collector=ZapiPerf:WorkloadVolume
2024-04-12T13:59:52+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=262 bytesRx=24094 collector=ZapiPerf:WorkloadVolume metrics=25 numCalls=1 pollMs=267 task=counter zBegin=1712910592236
2024-04-12T13:59:53+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=619 bytesRx=85422 collector=ZapiPerf:WorkloadVolume instances=188 numCalls=1 pollMs=620 task=instance zBegin=1712910592504
2024-04-12T13:59:58+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=1055 bytesRx=10621 calcMs=0 collector=ZapiPerf:Workload exportMs=1 instances=5 instancesExported=5 metrics=135 metricsExported=120 numCalls=2 numPartials=0 parseMs=11 pluginMs=1235 pollMs=2302 skips=0 zBegin=1712910596611
2024-04-12T14:00:08+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=965 bytesRx=10621 calcMs=0 collector=ZapiPerf:Workload exportMs=1 instances=5 instancesExported=5 metrics=135 metricsExported=120 numCalls=2 numPartials=0 parseMs=4 pluginMs=1196 pollMs=2166 skips=0 zBegin=1712910606616
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.cache property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.cloud_s2c property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.bamboo_ssd property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.hya_non_cache property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=sequential_reads_base key=sequential_reads property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.fc_miss property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.ext_cache property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.disk property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.pmem property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.hya_cache property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.cloud property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=read_io_type_base key=read_io_type.hya_hdd property=percent
2024-04-12T14:00:14+05:30 WRN zapiperf/zapiperf.go:819 > Base counter missing Poller=u2 collector=ZapiPerf:WorkloadVolume comment=sequential_writes_base key=sequential_writes property=percent
2024-04-12T14:00:14+05:30 INF collector/collector.go:595 > Collected Poller=u2 apiMs=1942 bytesRx=115821 calcMs=3 collector=ZapiPerf:WorkloadVolume exportMs=11 instances=188 instancesExported=188 metrics=2068 metricsExported=2068 numCalls=1 numPartials=0 parseMs=40 pluginMs=0 pollMs=1987 skips=4512 zBegin=1712910612190
cgrinds commented 5 months ago

Verified on 24.05.0 commit 74dcf38a, no errors or warnings during ZapiPerf polls