Templates not matched as they should

ybizeul commented 3 years ago

Describe the bug Some templates aren't match with current running version

Environment NAbox dev

Harvest version: [copy paste output of harvest version 21.05.4-2 (commit 19f8f25) (build date 2021-07-22T15:42:14+0000) linux/amd64]
Command line arguments used: /netapp-harvest/bin/poller -l 0 ...
OS: NAbox dev
Install method: native
ONTAP Version: 8.2.5P5 7-Mode

To Reproduce Not sure what the conditions are

Expected behavior Template should be matched, metrics collected

Actual behavior Template not matched

Here are a couple of example for a template that is found, and one that isn't, for the same system. As you can see, map[] is empty when template isn't matched.

Template found:

nabox-harvest2        | 10:01PM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:FcpPort
nabox-harvest2        | 10:01PM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:FcpPort
nabox-harvest2        | 10:01PM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:FcpPort
nabox-harvest2        | 10:01PM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: map[8.2.5:true] Poller=7mode collector=ZapiPerf:FcpPort
nabox-harvest2        | 10:01PM DBG goharvest2/cmd/poller/collector/helpers.go:106 > selected best-fitting subtemplate [conf/zapiperf/7mode/8.2.5/fcp.yaml] Poller=7mode collector=ZapiPerf:FcpPort

Template not found:

nabox-harvest2        | 10:01PM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:Volume
nabox-harvest2        | 10:01PM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:Volume
nabox-harvest2        | 10:01PM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: map[] Poller=7mode collector=ZapiPerf:Volume
nabox-harvest2        | 10:01PM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: volume.yaml error="No best-fitting subtemplate version found" Poller=7mode collector=ZapiPerf:Volume
nabox-harvest2        | 10:01PM WRN command-line-arguments/poller.go:532 > init collector-object (ZapiPerf:Volume): No best-fitting subtemplate version found Poller=7mode

cgrinds commented 3 years ago

thanks @ybizeul

There were two issues fixed a couple of weeks ago that address this problem.

The first problem is 7-mode returns a different version string than c-mode. The fix was to beef up our parsing for 7-mode versions and add more unit tests. https://github.com/NetApp/harvest/pull/381/commits/8d5bb19d0737c3a7453918acc6062389aee99726

The second problem was with template selection that impacted both 7-mode and later cmode version, especially > 9.10.X That fix landed https://github.com/NetApp/harvest/commit/32fc838d8f05a554c0b2cf00d15cf047e3a23974 and more unit tests added.

Can you try building from source and confirming that fixes this bug?

ybizeul commented 3 years ago

DIdn't go really well.

root@nabox-harvest2:/netapp-harvest/bin# ./harvest --version
harvest version 21.05.4-21.08.1519 (commit 089ca9f) (build date 2021-08-15T19:15:46+0200) linux/amd64
root@nabox-harvest2:/netapp-harvest/bin# ./poller -l 0 -p 7mode
4:59AM DBG goharvest2/pkg/logging/logger.go:116 > logging configured Poller=7mode consoleLoggingEnabled=true fileLogging=false fileName= logDirectory=/var/log/harvest/ loglevel=trace maxAgeInDays=30 maxBackups=10 maxSizeMB=10 prefixKey=Poller prefixValue=7mode
4:59AM INF command-line-arguments/poller.go:153 > log level used: trace Poller=7mode
4:59AM INF command-line-arguments/poller.go:154 > options config: /conf/harvest.yml Poller=7mode
4:59AM DBG command-line-arguments/poller.go:166 > * harvest version 21.05.4-21.08.1519 (commit 089ca9f) (build date 2021-08-15T19:15:47+0200) linux/amd64
 *s Poller=7mode
4:59AM DBG command-line-arguments/poller.go:167 > options= Poller= 7mode, Daemon = false, Debug = false, Profiling = 0, PromPort = 0, LogLevel = 0, HomePath = /conf/, LogPath = /var/log/harvest/, Config = /conf/harvest.yml, Hostname = nabox-harvest2, Version = 21.05.4 Poller=7mode
4:59AM DBG command-line-arguments/poller.go:173 > set signal handler for [hangup interrupt terminated quit] Poller=7mode
4:59AM INF command-line-arguments/poller.go:179 > started in foreground [pid=102] Poller=7mode
4:59AM DBG command-line-arguments/poller.go:183 > importing config [/conf/harvest.yml] Poller=7mode
4:59AM WRN command-line-arguments/poller.go:482 > Unable to load template. error="open /conf/conf/zapi/custom.yaml: no such file or directory" Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Node
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/node.yaml] for [8.2.5] Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status healthy 1 true map[true:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (system-info.memory-size) [system_memory_size] => [system-info memory-size] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.maximum-aggregate-size) as label [max_aggr_size] => [system-info maximum-aggregate-size] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.maximum-flexible-volume-count) as label [max_vol_num] => [system-info maximum-flexible-volume-count] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.maximum-flexible-volume-size) as label [max_vol_size] => [system-info maximum-flexible-volume-size] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.system-name) as label [node] => [system-info system-name] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (system-info.system-name) as instance key [node] => [system-info system-name] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.system-model) as label [model] => [system-info system-model] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.system-serial-number) as label [serial] => [system-info system-serial-number] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.vendor-id) as label [vendor] => [system-info vendor-id] Poller=7mode collector=Zapi:Node
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (system-info.product-version) as label [version] => [system-info product-version] Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 8 labels Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[system-info system-name]] Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [system-info] Poller=7mode collector=Zapi:Node
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Node
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Node) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Aggregate
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/aggr.yaml] for [8.2.5] Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status state 1 true map[online:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (aggr-info.name) as label [aggr] => [aggr-info name] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (aggr-info.uuid) as label [uuid] => [aggr-info uuid] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (aggr-info.uuid) as instance key [uuid] => [aggr-info uuid] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (aggr-info.home-name) as label [node] => [aggr-info home-name] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (aggr-info.type) as label [type] => [aggr-info type] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (aggr-info.state) as label [state] => [aggr-info state] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.disk-count) [raid_disk_count] => [aggr-info disk-count] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.plex-count) [raid_plex_count] => [aggr-info plex-count] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.raid-size) [raid_size] => [aggr-info raid-size] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.volume-count) [volume_count_flexvol] => [aggr-info volume-count] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.size-available) [space_available] => [aggr-info size-available] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.size-total) [space_total] => [aggr-info size-total] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.size-used) [space_used] => [aggr-info size-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.size-percentage-used) [space_used_percent] => [aggr-info size-percentage-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-files-private-used) [inode_files_private_used] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-files-private-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-files-total) [inode_files_total] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-files-total] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-files-used) [inode_files_used] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-files-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-inodefile-private-capacity) [inode_inodefile_private_capacity] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-inodefile-private-capacity] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-inodefile-public-capacity) [inode_inodefile_public_capacity] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-inodefile-public-capacity] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-maxfiles-available) [inode_maxfiles_available] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-maxfiles-available] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-maxfiles-possible) [inode_maxfiles_possible] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-maxfiles-possible] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-maxfiles-used) [inode_maxfiles_used] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-maxfiles-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-percent-inode-used-capacity) [inode_used_percent] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-percent-inode-used-capacity] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-hybrid-cache-size-total) [hybrid_cache_size] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-hybrid-cache-size-total] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-total-reserved-space) [space_reserved] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-total-reserved-space] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-sis-percent-saved) [space_sis_saved_percent] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-sis-percent-saved] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-sis-saved-space) [space_sis_saved] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-sis-saved-space] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.aggregate-space.fs-space-info.fs-sis-shared-space) [space_sis_shared_count] => [aggr-info aggregate-space-details aggregate-space-info aggregate-space fs-space-info fs-sis-shared-space] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-files-total) [snapshot_files_total] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-files-total] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-files-used) [snapshot_files_used] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-files-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-maxfiles-available) [snapshot_maxfiles_available] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-maxfiles-available] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-maxfiles-possible) [snapshot_maxfiles_possible] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-maxfiles-possible] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-maxfiles-used) [snapshot_maxfiles_used] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-maxfiles-used] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-percent-inode-used-capacity) [snapshot_inode_used_percent] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-percent-inode-used-capacity] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-percent-used-capacity) [snapshot_used_percent] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-percent-used-capacity] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-size-available) [snapshot_size_available] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-size-available] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-size-total) [snapshot_size_total] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-size-total] Poller=7mode collector=Zapi:Aggregate
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (aggr-info.aggregate-space-details.aggregate-space-info.snapshot-space.snapshot-space-info.snapshot-size-used) [snapshot_size_used] => [aggr-info aggregate-space-details aggregate-space-info snapshot-space snapshot-space-info snapshot-size-used] Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 5 labels Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[aggr-info uuid]] Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [aggr-info] Poller=7mode collector=Zapi:Aggregate
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Aggregate
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Aggregate) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Volume
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/volume.yaml] for [8.2.5] Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status state 1 true map[online:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (volume-info.name) as label [volume] => [volume-info name] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (volume-info.type) as label [style] => [volume-info type] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (volume-info.uuid) as label [uuid] => [volume-info uuid] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (volume-info.uuid) as instance key [uuid] => [volume-info uuid] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (volume-info.state) as label [state] => [volume-info state] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.containing-aggregate) [aggr] => [volume-info containing-aggregate] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.files-total) [files_total] => [volume-info files-total] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.files-used) [files_used] => [volume-info files-used] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.filesystem-size) [filesystem_size] => [volume-info filesystem-size] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.size-available) [size_available] => [volume-info size-available] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.size-total) [size_total] => [volume-info size-total] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.size-used) [size_used] => [volume-info size-used] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.percentage-used) [size_used_percent] => [volume-info percentage-used] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.autosize.grow-threshold-percent) [autosize_grow_threshold_percent] => [volume-info autosize grow-threshold-percent] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.autosize.maximum-size) [autosize_maximum_size] => [volume-info autosize maximum-size] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.compress-saved) [sis_compress_saved] => [volume-info sis compress-saved] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.dedup-saved) [sis_dedup_saved] => [volume-info sis dedup-saved] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.percent-compress-saved) [sis_compress_saved_percent] => [volume-info sis percent-compress-saved] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.percent-dedup-saved) [sis_dedup_saved_percent] => [volume-info sis percent-dedup-saved] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.percent-total-saved) [sis_total_saved_percent] => [volume-info sis percent-total-saved] Poller=7mode collector=Zapi:Volume
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (volume-info.sis.total-saved) [sis_total_saved] => [volume-info sis total-saved] Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 4 labels Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[volume-info uuid]] Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [volume-info] Poller=7mode collector=Zapi:Volume
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Volume
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Volume) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:SnapMirror
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:SnapMirror
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=Zapi:SnapMirror
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: snapmirror.yaml error="No best-fit template found" Poller=7mode collector=Zapi:SnapMirror
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (Zapi:SnapMirror): No best-fit template found Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Disk
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/disk.yaml] for [8.2.5] Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status offline 0 true map[false:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (disk-detail-info.bytes-per-sector) [bytes_per_sector] => [disk-detail-info bytes-per-sector] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.disk-model) as label [model] => [disk-detail-info disk-model] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.disk-type) as label [type] => [disk-detail-info disk-type] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.disk-uid) as label [detail_uid] => [disk-detail-info disk-uid] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (disk-detail-info.disk-uid) as instance key [detail_uid] => [disk-detail-info disk-uid] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.is-offline) as label [offline] => [disk-detail-info is-offline] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.name) as label [disk] => [disk-detail-info name] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (disk-detail-info.poweron-hours) [uptime] => [disk-detail-info poweron-hours] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (disk-detail-info.raw-disk-sectors) [sectors] => [disk-detail-info raw-disk-sectors] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.serial-number) as label [serial_number] => [disk-detail-info serial-number] Poller=7mode collector=Zapi:Disk
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (disk-detail-info.shelf) as label [shelf] => [disk-detail-info shelf] Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 7 labels Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[disk-detail-info disk-uid]] Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [disk-detail-info] Poller=7mode collector=Zapi:Disk
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Disk
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Disk) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Shelf
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/shelf.yaml] for [8.2.5] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:504 > loaded plugin [Shelf] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:53 > plugin connected! Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:99 > added instance key: (cooling-element-list) (cooling-element-info) [fan_id] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (cooling-element-list) (cooling-element-info) [location] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:112 > added metric: (cooling-element-list) (cooling-element-info) [rpm] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:117 > added data for [cooling-element-list] with 2 metrics Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:99 > added instance key: (current-sensor-list) (current-sensor-info) [sensor_id] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (current-sensor-list) (current-sensor-info) [location] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:112 > added metric: (current-sensor-list) (current-sensor-info) [reading] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:117 > added data for [current-sensor-list] with 2 metrics Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:99 > added instance key: (temp-sensor-list) (temp-sensor-info) [sensor_id] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:112 > added metric: (temp-sensor-list) (temp-sensor-info) [reading] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (temp-sensor-list) (temp-sensor-info) [high_critical] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (temp-sensor-list) (temp-sensor-info) [high_warning] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (temp-sensor-list) (temp-sensor-info) [temp_is_ambient] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (temp-sensor-list) (temp-sensor-info) [low_critical] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (temp-sensor-list) (temp-sensor-info) [low_warning] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:117 > added data for [temp-sensor-list] with 2 metrics Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:99 > added instance key: (voltage-sensor-list) (voltage-sensor-info) [sensor_id] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:112 > added metric: (voltage-sensor-list) (voltage-sensor-info) [reading] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (voltage-sensor-list) (voltage-sensor-info) [location] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:117 > added data for [voltage-sensor-list] with 2 metrics Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:99 > added instance key: (power-supply-list) (power-supply-info) [psu_id] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (power-supply-list) (power-supply-info) [part_number] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (power-supply-list) (power-supply-info) [serial] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (power-supply-list) (power-supply-info) [type] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (power-supply-list) (power-supply-info) [enabled] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:103 > added instance label: (power-supply-list) (power-supply-info) [location] Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:117 > added data for [power-supply-list] with 1 metrics Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/plugins/shelf/shelf.go:122 > initialized with data [5] objects Poller=7mode plugin=Zapi:Shelf
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status state 1 true map[normal:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 2 plugins Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (shelf-environ-channel-info.channel-name) as label [channel] => [shelf-environ-channel-info channel-name] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (shelf-environ-channel-info.channel-name) as instance key [channel] => [shelf-environ-channel-info channel-name] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.shelf-id) as label [shelf_id] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info shelf-id] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.shelf-id) as instance key [shelf_id] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info shelf-id] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.shelf-status) as label [state] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info shelf-status] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.shelf-type) as label [environ_channel_type] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info shelf-type] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.status-reads-attempted) [status_reads_attempted] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info status-reads-attempted] Poller=7mode collector=Zapi:Shelf
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (shelf-environ-channel-info.shelf-environ-shelf-list.shelf-environ-shelf-info.status-reads-failed) [status_reads_failed] => [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info status-reads-failed] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 4 labels Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[shelf-environ-channel-info channel-name] [shelf-environ-channel-info shelf-environ-shelf-list shelf-environ-shelf-info shelf-id]] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [shelf-environ-channel-info] Poller=7mode collector=Zapi:Shelf
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Shelf
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Shelf) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Status
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Status
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Status
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/status.yaml] for [8.2.5] Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status status 1 true map[ok:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Status
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (status) as label [status] => [status] Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 1 labels Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [] Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [status] Poller=7mode collector=Zapi:Status
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Status
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Status) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.6.0] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.6.0] Poller=7mode collector=Zapi:Subsystem
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapi/7mode/8.6.0/subsystem.yaml] for [8.2.5] Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status health 1 true map[ok:0]}] Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=Zapi:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:162 > Parsing counters: 1 values Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (diagnosis-subsystem-config-info.health) as label [health] => [diagnosis-subsystem-config-info health] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:120 > add (diagnosis-subsystem-config-info.subsystem) as label [subsystem] => [diagnosis-subsystem-config-info subsystem] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:125 > add (diagnosis-subsystem-config-info.subsystem) as instance key [subsystem] => [diagnosis-subsystem-config-info subsystem] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (diagnosis-subsystem-config-info.outstanding-alert-count) [outstanding_alerts] => [diagnosis-subsystem-config-info outstanding-alert-count] Poller=7mode collector=Zapi:Subsystem
4:59AM TRC goharvest2/cmd/collectors/zapi/collector/parsers.go:139 > add as metric (diagnosis-subsystem-config-info.suppressed-alert-count) [suppressed_alerts] => [diagnosis-subsystem-config-info suppressed-alert-count] Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:170 > initialized cache with 0 metrics and 2 labels Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:179 > Parsed Instance Keys: [[diagnosis-subsystem-config-info subsystem]] Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:180 > Parsed Instance Key Prefix: [diagnosis-subsystem-config-info] Poller=7mode collector=Zapi:Subsystem
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:73 > initialized Poller=7mode collector=Zapi:Subsystem
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (Zapi:Subsystem) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=Zapi:Lun
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapi/7mode] Poller=7mode collector=Zapi:Lun
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=Zapi:Lun
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: lun.yaml error="No best-fit template found" Poller=7mode collector=Zapi:Lun
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (Zapi:Lun): No best-fit template found Poller=7mode
4:59AM DBG command-line-arguments/poller.go:560 > initialized (Zapi) with 7 objects Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Node) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Aggregate) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Volume) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Disk) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Shelf) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Status) not available Poller=7mode
4:59AM TRC command-line-arguments/poller.go:571 > expName nabox-prometheus Poller=7mode
4:59AM DBG goharvest2/cmd/exporters/prometheus/prometheus.go:115 > using default cache_max_keep [180s] Poller=7mode exporter=nabox-prometheus
4:59AM ERR goharvest2/cmd/exporters/prometheus/prometheus.go:164 > Issue while reading prometheus port Poller=7mode exporter=nabox-prometheus
4:59AM ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Subsystem) not available Poller=7mode
4:59AM WRN command-line-arguments/poller.go:482 > Unable to load template. error="open /conf/conf/zapiperf/custom.yaml: no such file or directory" Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/system_node.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [name] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [node] Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= SystemNode --> node Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:SystemNode
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:SystemNode) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/hostadapter.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [name] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [hostadapter] Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= HostAdapter --> hostadapter Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:HostAdapter
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:HostAdapter) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:Path
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:Path
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:Path
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/path.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:129 > fields := [path .` hostadapter,target_wwpn] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:131 > fields = [. hostadapter,target_wwpn] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:135 > (split) parsed rule [{. path [hostadapter target_wwpn]}] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:129 > fields := [path _` hostadapter,target_wwpn] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:131 > fields = [_ hostadapter,target_wwpn] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:135 > (split) parsed rule [{_ path [hostadapter target_wwpn]}] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 2 rules for 1 actions Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [uuid] Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [path] Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= Path --> path Poller=7mode collector=ZapiPerf:Path
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:Path
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:Path) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:Disk
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:Disk
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:Disk
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/disk.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:129 > fields := [raid_group /` ,aggr,plex,raid] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:131 > fields = [/ ,aggr,plex,raid] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:135 > (split) parsed rule [{/ raid_group [ aggr plex raid]}] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 1 actions Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [Aggregator] Poller=7mode collector=ZapiPerf:Disk
4:59AM TRC goharvest2/cmd/poller/plugin/aggregator/aggregator.go:60 > parsing raw rule: [node] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM TRC goharvest2/cmd/poller/plugin/aggregator/aggregator.go:67 > handling first field: [node] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM DBG goharvest2/cmd/poller/plugin/aggregator/aggregator.go:107 > parsed rule [{node    <nil> [] false map[]}] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM TRC goharvest2/cmd/poller/plugin/aggregator/aggregator.go:60 > parsing raw rule: [aggr node] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM TRC goharvest2/cmd/poller/plugin/aggregator/aggregator.go:67 > handling first field: [aggr] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM TRC goharvest2/cmd/poller/plugin/aggregator/aggregator.go:99 > handling second field: [node] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM DBG goharvest2/cmd/poller/plugin/aggregator/aggregator.go:107 > parsed rule [{aggr    <nil> [node] false map[]}] Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM DBG goharvest2/cmd/poller/plugin/aggregator/aggregator.go:49 > parsed 2 aggregation rules Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 2 plugins Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [name] Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [disk] Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= Disk --> disk Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:Disk
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:Disk) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:FCVI
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:FCVI
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=ZapiPerf:FCVI
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: fcvi.yaml error="No best-fit template found" Poller=7mode collector=ZapiPerf:FCVI
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:FCVI): No best-fit template found Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/ext_cache_obj.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [uuid] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [flashcache] Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= ExtCacheObj --> flashcache Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:ExtCacheObj
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:ExtCacheObj) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:WAFL
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:WAFL
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:WAFL
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/wafl.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [uuid] Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [wafl] Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= WAFL --> wafl Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:WAFL
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:WAFL) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/wafl_hya_per_aggr.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:131 > using instance_key = [uuid] (default) Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [flashpool] Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= WAFLAggr --> flashpool Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:WAFLAggr
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:WAFLAggr) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/wafl_hya_sizer.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [uuid] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [flashpool] Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= WAFLSizer --> flashpool Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:WAFLSizer
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:WAFLSizer) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/nic_common.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:504 > loaded plugin [Nic] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [LabelAgent] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/parse_rules.go:411 > (value_mapping) parsed rule [{status state 1 true map[up:0]}] Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/plugin/label_agent/label_agent.go:49 > parsed 1 rules for 0 actions Poller=7mode plugin=ZapiPerf:LabelAgent
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 2 plugins Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [name] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [nic] Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= NicCommon --> nic Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:NicCommon
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:NicCommon) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/fcp.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:504 > loaded plugin [Fcp] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [uuid] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [fcp] Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= FcpPort --> fcp Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:FcpPort
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:FcpPort) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:ObjectStoreClient
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:ObjectStoreClient
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=ZapiPerf:ObjectStoreClient
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: object_store_client_op.yaml error="No best-fit template found" Poller=7mode collector=ZapiPerf:ObjectStoreClient
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:ObjectStoreClient): No best-fit template found Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/volume_node.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:131 > using instance_key = [uuid] (default) Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [volume] Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= VolumeNode --> volume Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:VolumeNode
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:VolumeNode) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/token_manager.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using instance_key = [name] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using batch_size = [500] (default) Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:152 > using latency_io_reqd = [10] (default) Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:128 > using object = [token] Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:118 > object= TokenManager --> token Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG goharvest2/cmd/collectors/zapiperf/zapiperf.go:100 > initialized Poller=7mode collector=ZapiPerf:TokenManager
4:59AM DBG command-line-arguments/poller.go:552 > initialized collector-object (ZapiPerf:TokenManager) Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:HeadroomCPU
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:HeadroomCPU
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=ZapiPerf:HeadroomCPU
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: resource_headroom_cpu.yaml error="No best-fit template found" Poller=7mode collector=ZapiPerf:HeadroomCPU
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:HeadroomCPU): No best-fit template found Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:HeadroomAggr
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:HeadroomAggr
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=ZapiPerf:HeadroomAggr
4:59AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: resource_headroom_aggr.yaml error="No best-fit template found" Poller=7mode collector=ZapiPerf:HeadroomAggr
4:59AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:HeadroomAggr): No best-fit template found Poller=7mode
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
4:59AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
4:59AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:Processor
4:59AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:Processor
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:71 > available version dir: [8.2.5] Poller=7mode collector=ZapiPerf:Processor
4:59AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 1 available versions: [8.2.5] Poller=7mode collector=ZapiPerf:Processor
4:59AM INF goharvest2/cmd/poller/collector/helpers.go:114 > best-fit template [/conf/conf/zapiperf/7mode/8.2.5/processor.yaml] for [8.2.5] Poller=7mode collector=ZapiPerf:Processor
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:500 > loaded built-in plugin [Aggregator] Poller=7mode collector=ZapiPerf:Processor
4:59AM DBG goharvest2/cmd/poller/plugin/aggregator/aggregator.go:49 > parsed 0 aggregation rules Poller=7mode plugin=ZapiPerf:Aggregator
4:59AM INF goharvest2/cmd/collectors/zapi/collector/zapi.go:131 > no zapi plugin found for node<>node_cpu Poller=7mode collector=ZapiPerf:Processor
4:59AM DBG goharvest2/cmd/poller/collector/collector.go:504 > loaded plugin [node<>node_cpu] Poller=7mode collector=ZapiPerf:Processor
4:59AM ERR command-line-arguments/poller.go:802 > Poller panicked runtime error: invalid memory address or nil pointer dereference error="goroutine panic => goroutine 1 [running]:\nruntime/debug.Stack(0xc0000b8d20, 0x94c103, 0x0)\n\truntime/debug/stack.go:24 +0x9f\nmain.main.func1()\n\tcommand-line-arguments/poller.go:802 +0xae\npanic(0x94c120, 0xd31920)\n\truntime/panic.go:965 +0x1b9\ngoharvest2/cmd/poller/collector.(*AbstractCollector).LoadPlugins(0xc0001dabb0, 0xc000221e60, 0xa8fa58, 0xc000150e00, 0x0, 0x0)\n\tgoharvest2/cmd/poller/collector/collector.go:507 +0x2f5\ngoharvest2/cmd/poller/collector.Init(0xa8fa58, 0xc000150e00, 0x0, 0x0)\n\tgoharvest2/cmd/poller/collector/collector.go:194 +0x1683\ngoharvest2/cmd/collectors/zapiperf.(*ZapiPerf).Init(0xc000150e00, 0xc0001dabb0, 0xc0001ff600, 0x9)\n\tgoharvest2/cmd/collectors/zapiperf/zapiperf.go:88 +0xa5\nmain.(*Poller).newCollector(0xc0000d2900, 0xc0000b33d8, 0x8, 0xc0001ff600, 0x9, 0xc000571cb0, 0xa8fa58, 0xc000145960, 0xa817a0, 0xc00016c950)\n\tcommand-line-arguments/poller.go:657 +0x34b\nmain.(*Poller).loadCollector(0xc0000d2900, 0xc0000b33d8, 0x8, 0xd32570, 0x0, 0x0, 0x0, 0x0)\n\tcommand-line-arguments/poller.go:539 +0x1165\nmain.(*Poller).Init(0xc0000d2900, 0xc0000d2900, 0x0)\n\tcommand-line-arguments/poller.go:249 +0xb88\nmain.startPoller(0xd3aac0, 0xc0000debc0, 0x0, 0x4)\n\tcommand-line-arguments/poller.go:763 +0x4f\ngithub.com/spf13/cobra.(*Command).execute(0xd3aac0, 0xc0000ba010, 0x4, 0x4, 0xd3aac0, 0xc0000ba010)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:860 +0x2c2\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xd3aac0, 0x401220, 0x0, 0x0)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974 +0x375\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main()\n\tcommand-line-arguments/poller.go:809 +0x53\n" Poller=7mode stack=[{"func":"New","line":"35","source":"errors.go"},{"func":"main.func1","line":"802","source":"poller.go"},{"func":"gopanic","line":"965","source":"panic.go"},{"func":"panicmem","line":"212","source":"panic.go"},{"func":"sigpanic","line":"734","source":"signal_unix.go"},{"func":"(*AbstractCollector).LoadPlugins","line":"507","source":"collector.go"},{"func":"Init","line":"194","source":"collector.go"},{"func":"(*ZapiPerf).Init","line":"88","source":"zapiperf.go"},{"func":"(*Poller).newCollector","line":"657","source":"poller.go"},{"func":"(*Poller).loadCollector","line":"539","source":"poller.go"},{"func":"(*Poller).Init","line":"249","source":"poller.go"},{"func":"startPoller","line":"763","source":"poller.go"},{"func":"(*Command).execute","line":"860","source":"command.go"},{"func":"(*Command).ExecuteC","line":"974","source":"command.go"},{"func":"(*Command).Execute","line":"902","source":"command.go"},{"func":"main","line":"809","source":"poller.go"},{"func":"main","line":"225","source":"proc.go"},{"func":"goexit","line":"1371","source":"asm_amd64.s"}]
4:59AM FTL command-line-arguments/poller.go:803 > (main) terminating abnormally, tip: run in foreground mode (with "--loglevel 0") to debug Poller=7mode

cgrinds commented 3 years ago

hey @ybizeul

Looks like the template issue you were hitting is fixed with HEAD, right?

Missing PromPort

One issue I noticed, (maybe was intentional), the promPort is missing which means the exporter won't start.

ERR command-line-arguments/poller.go:699 > init exporter (nabox-prometheus): missing parameter => port Poller=7mode
4:59AM WRN command-line-arguments/poller.go:576 > exporter (nabox-prometheus) requested by (Zapi:Aggregate) not available Poller=7mode

Panic

Looks like the collector panicked. The panic happened because a plugin failed to load because it was not found, which is unexpected.

These lines are interesting

a 4:59AM DBG collector.go:500 > loaded built-in plugin [Aggregator] Poller=7mode collector=ZapiPerf:Processor
b 4:59AM DBG aggregator.go:49 > parsed 0 aggregation rules Poller=7mode plugin=ZapiPerf:Aggregator
c 4:59AM INF zapi.go:131 > no zapi plugin found for node<>node_cpu Poller=7mode collector=ZapiPerf:Processor
d 4:59AM DBG collector.go:504 > loaded plugin [node<>node_cpu] Poller=7mode collector=ZapiPerf:Processor

line a loads the built-in Aggregator when initing ZapiPerf:Processor that's good and expected

line b though is unexpected since it says there are zero aggregation rules. That makes me wonder if there is a problem with your conf/zapiperf/7mode/8.2.5/processor.yaml

I expect something like this, which you have earlier in the log.

3:35PM DBG cmd/poller/collector/collector.go:500 > loaded built-in plugin [Aggregator] Poller=venkat collector=ZapiPerf:Processor
3:36PM TRC cmd/poller/plugin/aggregator/aggregator.go:60 > parsing raw rule: [node<>node_cpu] Poller=venkat plugin=ZapiPerf:Aggregator
3:36PM TRC cmd/poller/plugin/aggregator/aggregator.go:67 > handling first field: [node<>node_cpu] Poller=venkat plugin=ZapiPerf:Aggregator
3:36PM DBG cmd/poller/plugin/aggregator/aggregator.go:107 > parsed rule [{node node_cpu   <nil> [] false map[]}] Poller=venkat plugin=ZapiPerf:Aggregator
3:36PM DBG cmd/poller/plugin/aggregator/aggregator.go:47 > parsed 1 aggregation rule Poller=venkat plugin=ZapiPerf:Aggregator
3:36PM DBG cmd/poller/collector/collector.go:513 > initialized 1 plugins Poller=venkat collector=ZapiPerf:Processor

Line 503 attempted the load, but failed, and printed the message above returning a nil

The nil pointer dereference happens a few lines down at 507

There should a nil check at 504 or 507 that returns an init failed error, but that doesn't explain why you're seeing this.

When I run with 089ca9fd3c5ca2932db5190f03f90a6859448541 I don't hit this problem when loading the processor.yaml template, but I'm running against a cdot system and not 7-mode. Diffing the 7mode and cmode template though doesn't turn up anything interesting.

Did you modify processor.yaml or move any of the templates? Please paste the following?

find conf

conf
conf/zapiperf
conf/zapiperf/7mode
conf/zapiperf/7mode/8.2.5
conf/zapiperf/7mode/8.2.5/fcp.yaml
conf/zapiperf/7mode/8.2.5/path.yaml
conf/zapiperf/7mode/8.2.5/nfsv3_node.yaml
conf/zapiperf/7mode/8.2.5/iscsi_lif.yaml
conf/zapiperf/7mode/8.2.5/token_manager.yaml
conf/zapiperf/7mode/8.2.5/nic_common.yaml
conf/zapiperf/7mode/8.2.5/nfsv4_node.yaml
conf/zapiperf/7mode/8.2.5/cifs_node.yaml
conf/zapiperf/7mode/8.2.5/processor.yaml
conf/zapiperf/7mode/8.2.5/system_node.yaml
conf/zapiperf/7mode/8.2.5/volume_node.yaml
conf/zapiperf/7mode/8.2.5/hostadapter.yaml
conf/zapiperf/7mode/8.2.5/lun.yaml
conf/zapiperf/7mode/8.2.5/copy_manager.yaml
conf/zapiperf/7mode/8.2.5/wafl_hya_sizer.yaml
conf/zapiperf/7mode/8.2.5/wafl.yaml
conf/zapiperf/7mode/8.2.5/wafl_hya_per_aggr.yaml
conf/zapiperf/7mode/8.2.5/ext_cache_obj.yaml
conf/zapiperf/7mode/8.2.5/disk.yaml
conf/zapiperf/cdot
conf/zapiperf/cdot/9.8.0
conf/zapiperf/cdot/9.8.0/nfsv4.yaml
conf/zapiperf/cdot/9.8.0/fcp_lif.yaml
conf/zapiperf/cdot/9.8.0/fcp.yaml
conf/zapiperf/cdot/9.8.0/nfsv4_1_node.yaml
conf/zapiperf/cdot/9.8.0/path.yaml
conf/zapiperf/cdot/9.8.0/nfsv3_node.yaml
conf/zapiperf/cdot/9.8.0/iscsi_lif.yaml
conf/zapiperf/cdot/9.8.0/token_manager.yaml
conf/zapiperf/cdot/9.8.0/volume.yaml
conf/zapiperf/cdot/9.8.0/nic_common.yaml
conf/zapiperf/cdot/9.8.0/wafl_comp_aggr_vol_bin.yaml
conf/zapiperf/cdot/9.8.0/nfsv4_node.yaml
conf/zapiperf/cdot/9.8.0/cifs_node.yaml
conf/zapiperf/cdot/9.8.0/processor.yaml
conf/zapiperf/cdot/9.8.0/system_node.yaml
conf/zapiperf/cdot/9.8.0/nfsv4_1.yaml
conf/zapiperf/cdot/9.8.0/volume_node.yaml
conf/zapiperf/cdot/9.8.0/lif.yaml
conf/zapiperf/cdot/9.8.0/object_store_client_op.yaml
conf/zapiperf/cdot/9.8.0/fcvi.yaml
conf/zapiperf/cdot/9.8.0/workload.yaml
conf/zapiperf/cdot/9.8.0/nfsv3.yaml
conf/zapiperf/cdot/9.8.0/hostadapter.yaml
conf/zapiperf/cdot/9.8.0/lun.yaml
conf/zapiperf/cdot/9.8.0/copy_manager.yaml
conf/zapiperf/cdot/9.8.0/qtree.yaml
conf/zapiperf/cdot/9.8.0/wafl_hya_sizer.yaml
conf/zapiperf/cdot/9.8.0/resource_headroom_cpu.yaml
conf/zapiperf/cdot/9.8.0/wafl.yaml
conf/zapiperf/cdot/9.8.0/cifs_vserver.yaml
conf/zapiperf/cdot/9.8.0/wafl_hya_per_aggr.yaml
conf/zapiperf/cdot/9.8.0/ext_cache_obj.yaml
conf/zapiperf/cdot/9.8.0/workload_detail_volume.yaml
conf/zapiperf/cdot/9.8.0/workload_detail.yaml
conf/zapiperf/cdot/9.8.0/workload_volume.yaml
conf/zapiperf/cdot/9.8.0/disk.yaml
conf/zapiperf/cdot/9.8.0/resource_headroom_aggr.yaml
conf/zapiperf/cdot/9.10.1
conf/zapiperf/cdot/9.10.1/fcp.yaml
conf/zapiperf/cdot/9.10.1/namespace.yaml
conf/zapiperf/cdot/9.10.1/nvmf_lif.yaml
conf/zapiperf/default.yaml
conf/zapi
conf/zapi/7mode
conf/zapi/7mode/8.6.0
conf/zapi/7mode/8.6.0/volume.yaml
conf/zapi/7mode/8.6.0/subsystem.yaml
conf/zapi/7mode/8.6.0/aggr.yaml
conf/zapi/7mode/8.6.0/status.yaml
conf/zapi/7mode/8.6.0/shelf.yaml
conf/zapi/7mode/8.6.0/node.yaml
conf/zapi/7mode/8.6.0/disk.yaml
conf/zapi/cdot
conf/zapi/cdot/9.8.0
conf/zapi/cdot/9.8.0/volume.yaml
conf/zapi/cdot/9.8.0/subsystem.yaml
conf/zapi/cdot/9.8.0/aggr.yaml
conf/zapi/cdot/9.8.0/lun.yaml
conf/zapi/cdot/9.8.0/snapmirror.yaml
conf/zapi/cdot/9.8.0/status.yaml
conf/zapi/cdot/9.8.0/shelf.yaml
conf/zapi/cdot/9.8.0/node.yaml
conf/zapi/cdot/9.8.0/disk.yaml
conf/zapi/default.yaml
conf/simple
conf/simple/default.yaml
conf/unix
conf/unix/default.yaml

cat conf/zapiperf/7mode/8.2.5/processor.yaml

name:                     Processor
query:                    processor
object:                   processor

instance_key:             name

counters:
  - node_name             => node
  - instance_name
  - domain_busy
  - processor_busy

plugins:
  Aggregator:
    - node<>node_cpu

# only export node-level averages from plugin
# set this true or comment, to get data for each cpu
export_data: false

export_options:
  instance_keys:
    - cpus

ybizeul commented 3 years ago

Ok I was hoping the panic was caused by missing --promPort, but it's not, same behavior if I add it.

8:29AM ERR command-line-arguments/poller.go:802 > Poller panicked runtime error: invalid memory address or nil pointer dereference error="goroutine panic => goroutine 1 [running]:\nruntime/debug.Stack(0xc000132d80, 0x94c103, 0x0)\n\truntime/debug/stack.go:24 +0x9f\nmain.main.func1()\n\tcommand-line-arguments/poller.go:802 +0xae\npanic(0x94c120, 0xd31920)\n\truntime/panic.go:965 +0x1b9\ngoharvest2/cmd/poller/collector.(*AbstractCollector).LoadPlugins(0xc0000f0630, 0xc0004f9290, 0xa8fa58, 0xc0002e9490, 0x0, 0x0)\n\tgoharvest2/cmd/poller/collector/collector.go:507 +0x2f5\ngoharvest2/cmd/poller/collector.Init(0xa8fa58, 0xc0002e9490, 0x0, 0x0)\n\tgoharvest2/cmd/poller/collector/collector.go:194 +0x1683\ngoharvest2/cmd/collectors/zapiperf.(*ZapiPerf).Init(0xc0002e9490, 0xc0000f0630, 0xc000362ee0, 0x9)\n\tgoharvest2/cmd/collectors/zapiperf/zapiperf.go:88 +0xa5\nmain.(*Poller).newCollector(0xc00014c900, 0xc00012d578, 0x8, 0xc000362ee0, 0x9, 0xc0001fb830, 0xa8fa58, 0xc0001bb5e0, 0xa817a0, 0xc0002d68d0)\n\tcommand-line-arguments/poller.go:657 +0x34b\nmain.(*Poller).loadCollector(0xc00014c900, 0xc00012d578, 0x8, 0xd32570, 0x0, 0x0, 0x0, 0x0)\n\tcommand-line-arguments/poller.go:539 +0x1165\nmain.(*Poller).Init(0xc00014c900, 0xc00014c900, 0x0)\n\tcommand-line-arguments/poller.go:249 +0xb88\nmain.startPoller(0xd3aac0, 0xc000132ba0, 0x0, 0x6)\n\tcommand-line-arguments/poller.go:763 +0x4f\ngithub.com/spf13/cobra.(*Command).execute(0xd3aac0, 0xc000134010, 0x6, 0x6, 0xd3aac0, 0xc000134010)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:860 +0x2c2\ngithub.com/spf13/cobra.(*Command).ExecuteC(0xd3aac0, 0x401220, 0x0, 0x0)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:974 +0x375\ngithub.com/spf13/cobra.(*Command).Execute(...)\n\tgithub.com/spf13/cobra@v1.2.1/command.go:902\nmain.main()\n\tcommand-line-arguments/poller.go:809 +0x53\n" Poller=7mode stack=[{"func":"New","line":"35","source":"errors.go"},{"func":"main.func1","line":"802","source":"poller.go"},{"func":"gopanic","line":"965","source":"panic.go"},{"func":"panicmem","line":"212","source":"panic.go"},{"func":"sigpanic","line":"734","source":"signal_unix.go"},{"func":"(*AbstractCollector).LoadPlugins","line":"507","source":"collector.go"},{"func":"Init","line":"194","source":"collector.go"},{"func":"(*ZapiPerf).Init","line":"88","source":"zapiperf.go"},{"func":"(*Poller).newCollector","line":"657","source":"poller.go"},{"func":"(*Poller).loadCollector","line":"539","source":"poller.go"},{"func":"(*Poller).Init","line":"249","source":"poller.go"},{"func":"startPoller","line":"763","source":"poller.go"},{"func":"(*Command).execute","line":"860","source":"command.go"},{"func":"(*Command).ExecuteC","line":"974","source":"command.go"},{"func":"(*Command).Execute","line":"902","source":"command.go"},{"func":"main","line":"809","source":"poller.go"},{"func":"main","line":"225","source":"proc.go"},{"func":"goexit","line":"1371","source":"asm_amd64.s"}]
8:29AM FTL command-line-arguments/poller.go:803 > (main) terminating abnormally, tip: run in foreground mode (with "--loglevel 0") to debug Poller=7mode


root@nabox-harvest2:/netapp-harvest/bin# find /conf/conf -type f
/conf/conf/zapiperf/default.yaml
/conf/conf/zapiperf/cdot/9.8.0/copy_manager.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv4_1.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv4.yaml
/conf/conf/zapiperf/cdot/9.8.0/ext_cache_obj.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv3.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv3_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/wafl.yaml
/conf/conf/zapiperf/cdot/9.8.0/token_manager.yaml
/conf/conf/zapiperf/cdot/9.8.0/lif.yaml
/conf/conf/zapiperf/cdot/9.8.0/nic_common.yaml
/conf/conf/zapiperf/cdot/9.8.0/cifs_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/volume_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/wafl_hya_sizer.yaml
/conf/conf/zapiperf/cdot/9.8.0/fcvi.yaml
/conf/conf/zapiperf/cdot/9.8.0/resource_headroom_aggr.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv4_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/hostadapter.yaml
/conf/conf/zapiperf/cdot/9.8.0/disk.yaml
/conf/conf/zapiperf/cdot/9.8.0/wafl_comp_aggr_vol_bin.yaml
/conf/conf/zapiperf/cdot/9.8.0/fcp_lif.yaml
/conf/conf/zapiperf/cdot/9.8.0/system_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/fcp.yaml
/conf/conf/zapiperf/cdot/9.8.0/iscsi_lif.yaml
/conf/conf/zapiperf/cdot/9.8.0/wafl_hya_per_aggr.yaml
/conf/conf/zapiperf/cdot/9.8.0/object_store_client_op.yaml
/conf/conf/zapiperf/cdot/9.8.0/processor.yaml
/conf/conf/zapiperf/cdot/9.8.0/path.yaml
/conf/conf/zapiperf/cdot/9.8.0/cifs_vserver.yaml
/conf/conf/zapiperf/cdot/9.8.0/resource_headroom_cpu.yaml
/conf/conf/zapiperf/cdot/9.8.0/volume.yaml
/conf/conf/zapiperf/cdot/9.8.0/nfsv4_1_node.yaml
/conf/conf/zapiperf/cdot/9.8.0/lun.yaml
/conf/conf/zapiperf/7mode/8.2.5/copy_manager.yaml
/conf/conf/zapiperf/7mode/8.2.5/ext_cache_obj.yaml
/conf/conf/zapiperf/7mode/8.2.5/nfsv3_node.yaml
/conf/conf/zapiperf/7mode/8.2.5/wafl.yaml
/conf/conf/zapiperf/7mode/8.2.5/token_manager.yaml
/conf/conf/zapiperf/7mode/8.2.5/nic_common.yaml
/conf/conf/zapiperf/7mode/8.2.5/cifs_node.yaml
/conf/conf/zapiperf/7mode/8.2.5/volume_node.yaml
/conf/conf/zapiperf/7mode/8.2.5/wafl_hya_sizer.yaml
/conf/conf/zapiperf/7mode/8.2.5/nfsv4_node.yaml
/conf/conf/zapiperf/7mode/8.2.5/hostadapter.yaml
/conf/conf/zapiperf/7mode/8.2.5/disk.yaml
/conf/conf/zapiperf/7mode/8.2.5/system_node.yaml
/conf/conf/zapiperf/7mode/8.2.5/fcp.yaml
/conf/conf/zapiperf/7mode/8.2.5/iscsi_lif.yaml
/conf/conf/zapiperf/7mode/8.2.5/wafl_hya_per_aggr.yaml
/conf/conf/zapiperf/7mode/8.2.5/processor.yaml
/conf/conf/zapiperf/7mode/8.2.5/path.yaml
/conf/conf/zapiperf/7mode/8.2.5/lun.yaml
/conf/conf/unix/default.yaml
/conf/conf/zapi/default.yaml
/conf/conf/zapi/cdot/9.8.0/aggr.yaml
/conf/conf/zapi/cdot/9.8.0/subsystem.yaml
/conf/conf/zapi/cdot/9.8.0/shelf.yaml
/conf/conf/zapi/cdot/9.8.0/disk.yaml
/conf/conf/zapi/cdot/9.8.0/status.yaml
/conf/conf/zapi/cdot/9.8.0/node.yaml
/conf/conf/zapi/cdot/9.8.0/snapmirror.yaml
/conf/conf/zapi/cdot/9.8.0/volume.yaml
/conf/conf/zapi/cdot/9.8.0/lun.yaml
/conf/conf/zapi/7mode/8.6.0/aggr.yaml
/conf/conf/zapi/7mode/8.6.0/subsystem.yaml
/conf/conf/zapi/7mode/8.6.0/shelf.yaml
/conf/conf/zapi/7mode/8.6.0/disk.yaml
/conf/conf/zapi/7mode/8.6.0/status.yaml
/conf/conf/zapi/7mode/8.6.0/node.yaml
/conf/conf/zapi/7mode/8.6.0/volume.yaml
/conf/conf/cars/default.yaml

root@nabox-harvest2:/netapp-harvest/bin# cat /conf/conf/zapiperf/7mode/8.2.5/processor.yaml

name:                     Processor
query:                    processor
object:                   processor

instance_key:             name

counters:
  - node_name             => node
  - instance_name
  - domain_busy
  - processor_busy

plugins:
  Aggregator:
  - node<>node_cpu

# only export node-level averages from plugin
# set this true or comment, to get data for each cpu
export_data: false

export_options:
  instance_keys:
    - cpus

ybizeul commented 3 years ago

There is an indentation difference, it it syntactically correct, but the parser doesn't seem to accept it. That comes from an earlier version of Harvest.

I'll make sure to clean up those templates so a single set is used.

I'm not exactly sure how to handle it. What I'll probably do is when installing harvest 2, always overwrite default template files and try and preserve user files in case they added custom templates.

I still have template issues though, I think they didn't appear before because of the crash.

./poller -p 7mode --promPort 1000 -l 0
[...]
8:52AM DBG goharvest2/cmd/poller/collector/helpers.go:61 > Looking for best-fitting template in [/conf/conf/zapiperf/7mode] Poller=7mode collector=ZapiPerf:Volume
8:52AM TRC goharvest2/cmd/poller/collector/helpers.go:82 > checking for 0 available versions: [] Poller=7mode collector=ZapiPerf:Volume
8:52AM ERR goharvest2/cmd/collectors/zapi/collector/zapi.go:99 > Error importing subtemplate: volume.yaml error="No best-fit template found" Poller=7mode collector=ZapiPerf:Volume
8:52AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:Volume): No best-fit template found Poller=7mode
8:52AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:58 > using default API version [1.3] Poller=7mode Zapi=Client
8:52AM DBG goharvest2/pkg/api/ontapi/zapi/client.go:137 > using default timeout [10] s Poller=7mode Zapi=Client
8:52AM DBG goharvest2/cmd/collectors/zapi/collector/zapi.go:88 > connected to: 7mode (7MODE version 8.2.5) (serial 4082368508) (NetApp Release 8.2.5P5 7-Mode: Thu Jan  7 04:22:14 PST 2021) Poller=7mode collector=ZapiPerf:LUN
^C
root@nabox-harvest2:/netapp-harvest/bin# ./harvest --version
harvest version 21.05.4-21.08.1519 (commit 089ca9f) (build date 2021-08-15T19:15:46+0200) linux/amd64

rahulguptajss commented 3 years ago

@ybizeul I think all related "No best-fit template found" are due to missing yaml files under 7 mode conf. https://github.com/NetApp/harvest/tree/main/conf/zapiperf/7mode/8.2.5

for example below warn happens as there is no volume.yaml available under 7 mode conf. 8:52AM WRN command-line-arguments/poller.go:545 > init collector-object (ZapiPerf:Volume): No best-fit template found Poller=7mode

ybizeul commented 3 years ago

@rahulguptajss that seems to be making sense.

My original concern is not having much metrics for the 7-mode systems.

rahulguptajss commented 3 years ago

@ybizeul I am working on this.

rahulguptajss commented 3 years ago

@ybizeul could you check if volume and node dashboard has data now for 7 mode ? I have opened an issue #423 for other identified issues.

ybizeul commented 3 years ago

Thanks @rahulguptajss, it seems to be working indeed with 21.05.4-21.08.2009. Closing this !

cgrinds commented 3 years ago

verified node and volume dashboards work with cluster running 8.2.5

NetApp / harvest

Templates not matched as they should #415

Missing PromPort

Panic