Closed db-wally007 closed 3 months ago
I just checked our 2 clusters and issue is a bit strange and maybe it will help you to replicate the issue.
1, TROY appliance - has volume with VolumeAnalytics feature enabled = no errors from the poller and VolumeAnalytics metrics are present in the Prometheus/Grafana
2, AGORA appliance - has no volume with VolumeAnalytics feature enabled but we have SVM-DR relationship setup from TROY -> to -> AGORA (i.e. SVM from TROY that has the VolumeAnalytics enabled is replicated to AGORA) = error in the logs as described in the earlier post.
It appears that the AGORA appliance has volumes with analytics enabled. Could you run the following command for the AGORA appliance? Please note that the USERNAME, PASSWORD, URL should be replaced with the appropriate credentials.
curl -s -k -u USERNAME:PASSWORD 'https://URL/api/storage/volumes?return_records=true&fields=name,svm.name,uuid&analytics.state=on&max_records=20&order_by=space.used%20desc&ignore_unknown_fields=true'
If there are records returned, could you take the UUID of a volume and replace UUID
in the following command to check if percentages are available in the response? It seems they are empty resulting in error log.
curl -s -k -u USERNAME:PASSWORD 'https://URL/api/storage/volumes/UUID/files?return_records=true&fields=analytics.by_accessed_time.bytes_used.percentages&type=directory&max_records=100'
Also, could you also share the ONTAP version of the Troy appliance?
You can also share output of these curl commands with us via ng-harvest-files@netapp.com
Email sent with output of the curl commands from both SVM-DR source (TROY) and SVM-DR destination (AGORA)
It includes home directories with internal usernames - which might be somewhat sensitive. Let me know if you received it.
TROY (AFF-250) has the exact same version as AGORA (AFF-220) (we were advised by the Netapp techs to try to keep ONTAP versions in sync for best results with SVM-DR functionality)
Thanks @db-wally007 we got the curl emails
hi @db-wally007 your curl request shows that ONTAP believes analytics are enabled for two volumes on agora, both of those volumes are on the same SVM. It sounds like you didn't think volume analytics were enabled on agora. Is that correct?
Interestingly, even though those two volumes say they have volume analytics enabled, when queried for analytics they return no analytics. 😄 I wonder if this is an ONTAP bug related to the SVM-DR relationship you mentioned? Are you on Discord? If so, please ask if this is a bug on the #ontap
channel.
Regardless, Harvest needs to do a better job ignoring these faux volume-analytics-enabled volumes. We'll fix that.
hi @db-wally007 your curl request shows that ONTAP believes analytics are enabled for two volumes on agora, both of those volumes are on the same SVM. It sounds like you didn't think volume analytics were enabled on agora. Is that correct?
Interestingly, even though those two volumes say they have volume analytics enabled, when queried for analytics they return no analytics. 😄 I wonder if this is an ONTAP bug related to the SVM-DR relationship you mentioned? Are you on Discord? If so, please ask if this is a bug on the
#ontap
channel.Regardless, Harvest needs to do a better job ignoring these faux volume-analytics-enabled volumes. We'll fix that.
VolumeAnalytics is enabled on volumes that reside on TROY (plato_svm) and that SVM gets replicated to AGORA via SVM-DR feature into dr_plato_svm SVM.
dr_plato_svm on AGORA is stopped, maybe that is the issue ? (while source SVM plato_svm is running, destination SVM dr_plato_svm must be stopped, obviously)
AGORA serves as a SVM-DR destination.
Re: Discord, (un)fortunately I do not (no social media accounts)
@db-wally007 It will be fixed via #3026.
In the meantime, you may try to disable volume activity tracking for these volumes which are in an SVM-DR stopped vserver.
To list the relevant volumes with activity tracking enabled, use the following command:
volume show -activity-tracking-state on -vserver
To disable activity tracking, use the following command:
volume activity-tracking off -volume
Ideally, in a DR relationship SVM, activity tracking should be disabled. I tried enabling it in version 9.15 but encountered the following error:
Volume activity tracking wasn't enabled on volume "umeng_aff300_05_svm5_root" in storage VM "xyz" due to the following reason: "This operation is not permitted on a Vserver that is configured as the destination for Vserver DR."
@db-wally007 It will be fixed via #3026.
In the meantime, you may try to disable volume activity tracking for these volumes which are in an SVM-DR stopped vserver.
Ideally, in a DR relationship SVM, activity tracking should be disabled. I tried enabling it in version 9.15 but encountered the following error:
Like I wrote a few times, VolumeAnalytics IS off on AGORA. The problem is with the query or query response that Harvest is doing.
Here is ONTAP output:
AGORA::*> volume activity-tracking show
vserver volume state
------- ------------------------------------------ -----
AGORA MDV_CRS_e26ccfc6e20911ea9b9cd039ea1acb75_A off
AGORA MDV_CRS_e26ccfc6e20911ea9b9cd039ea1acb75_B off
AGORA-01
vol0 off
AGORA-02
vol0 off
agora_svm
agora_svm_root_volume off
agora_svm
vmware_backup_agora off
dr_plato_svm
backup off
dr_plato_svm
home_dirs off
dr_plato_svm
pcap off
dr_plato_svm
plato_svm_root_volume off
dr_plato_svm
rdf off
dr_plato_svm
shared_data off
dr_plato_svm
xml off
13 entries were displayed.
AGORA::*>
and
AGORA::*> volume show -activity-tracking-state on
There are no entries matching your query.
AGORA::*>
Understood. Activity Tracking is used for Top K metrics in ONTAP. I should have asked you to check the output of the following CLI, as these are what we filter on in the REST call.
You should see some results for this SVM with the following query:
volume show -analytics-state on
However, we get empty records when we call the following REST curl, which makes sense as files were never accessed on this volume on the SVM-DR site yet. This is something we need to handle in Harvest. It will be fixed via #3026. Thanks.
curl -s -k -u xxx:xxx 'https://xxxx/api/storage/volumes/xxxx/files?return_records=true&fields=analytics.by_accessed_time.bytes_used.percentages&type=directory&max_records=100'
Understood. Activity Tracking is used for Top K metrics in ONTAP. I should have asked you to check the output of the following CLI, as these are what we filter on in the REST call.
You should see some results for this SVM with the following query:
volume show -analytics-state on
Here is the output from SVM-DR destination:
AGORA::*> volume show -analytics-state on
There are no entries matching your query.
AGORA::*>
and here is the output from SVM-DR source:
TROY::*> volume show -analytics-state on
Vserver Volume Aggregate State Type Size Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
plato_svm home_dirs - online RW 42.10TB 21.78TB 0%
plato_svm shared_data - online RW 42.10TB 21.89TB 0%
2 entries were displayed.
TROY::*>
verified in 24.08
A note for the community
Problem
Hello,
I've upgraded Harvest from 24.02 to the latest (as of today) 24.05 and upon container start, I see the following error flooding container log output (journald in my case)
Jun 29 17:00:28 podman1 netapp-harvest-exporter[3241772]: 2024-06-29T17:00:28Z ERR volumeanalytics/volumeanalytics.go:253 > set metric error="strconv.ParseFloat: parsing \"\": invalid syntax" Poller=agora object=VolumeAnalytics plugin=Rest:VolumeAnalytics value=
config file:
Container file:
harvest_entrypoint.sh:
Configuration
No response
Poller
agora poller
Version
Poller logs
No response
OS and platform
RHEL 9.4 , podman
ONTAP or StorageGRID version
NetApp Release 9.13.1P7
Additional Context
References
No response