galexrt / dellhw_exporter

Prometheus exporter for Dell Hardware components using Dell OMSA.
https://dellhw-exporter.galexrt.moe
Apache License 2.0
119 stars 41 forks source link

storage_vdisk parsing error #106

Closed nexix closed 7 months ago

nexix commented 8 months ago
dellhw_exporter, version 1.13.10 (branch: HEAD, revision: 03ef74fb6af9a63035d5c90e990be845080159b6)
  build user:       root@7c44df39d577
  build date:       20240228-22:12:19
  go version:       go1.21.7
  platform:         linux/amd64
  tags:             netgo static_build
Mar 21 15:53:57 m0394125.ppops.net dellhw_exporter[117188]: time="2024-03-21T15:53:57Z" level=error msg="storage_vdisk collector failed after 0.039089s: strconv.ParseFloat: parsing \"\": invalid syntax"
Mar 21 15:54:08 m0394125.ppops.net dellhw_exporter[117188]: time="2024-03-21T15:54:08Z" level=error msg="storage_vdisk collector failed after 0.035442s: strconv.ParseFloat: parsing \"\": invalid syntax"

Seeing parsing errors for vdisk on 2 of the newest nodes in our infrastructure. Let me know if I can provide any other information that would help.

galexrt commented 8 months ago

@nexix Please post the output of omreport storage vdisk -fmt ssv from the new node.

nexix commented 8 months ago

I think this is a false alarm and no changes are necessary. We believe the problem was an initial install of outdated OMSA tools on Alma Linux 9. We have updated the tools and no longer see the error on the exporter. Thank you for the reply and sorry for the bother!

galexrt commented 8 months ago

@nexix I see, do you have the version numbers from before and after updates? Maybe others with similar versions can use it as a guidance to update their OMSA tools :smiley:

nexix commented 8 months ago

We went back and looked at this to add details about version numbers and could not recreate the issue. In doing some more research we have noticed that when the disk is running through background initialization we are seeing the parsing errors and we conflated it with updating packages.

omreport storage vdisk -fmt ssv
List of Virtual Disks in the System

Controller PERC H730P Mini (Embedded)

ID;Status;Name;State;Hot Spare Policy violated;Encrypted;Progress;Layout;Size;T10 Protection Information Status;Associated Fluid Cache State ;Device Name;Bus Protocol;Media;Read Policy;Write Policy;Cache Policy;Strip Element Size;Disk Cache Policy
0;Ok;Virtual Disk0;Ready;Not Assigned;No;Not Applicable;RAID-6;93.13 GiB (99999547392 bytes);No;Not Applicable;/dev/sdb;SATA;HDD;Adaptive Read Ahead;Write Back;Not Applicable;128 KB;Unchanged
1;Ok;Virtual Disk1;Background Initialization;Not Assigned;No;98% complete;RAID-6;3,631.87 GiB (3899688747008 bytes);No;Not Applicable;/dev/sda;SATA;HDD;Adaptive Read Ahead;Write Back;Not Applicable;128 KB;Unchanged

When the initialization completes then the parsing errors stop.

omreport storage vdisk -fmt ssv
List of Virtual Disks in the System

Controller PERC H730P Mini (Embedded)

ID;Status;Name;State;Hot Spare Policy violated;Encrypted;Layout;Size;T10 Protection Information Status;Associated Fluid Cache State ;Device Name;Bus Protocol;Media;Read Policy;Write Policy;Cache Policy;Strip Element Size;Disk Cache Policy
0;Ok;Virtual Disk0;Ready;Not Assigned;No;RAID-6;93.13 GiB (99999547392 bytes);No;Not Applicable;/dev/sdb;SATA;HDD;Adaptive Read Ahead;Write Back;Not Applicable;128 KB;Unchanged
1;Ok;Virtual Disk1;Ready;Not Assigned;No;RAID-6;3,631.87 GiB (3899688747008 bytes);No;Not Applicable;/dev/sda;SATA;HDD;Adaptive Read Ahead;Write Back;Not Applicable;128 KB;Unchanged

Last reports are right at the end of initialization then it stops.

Mar 27 19:58:31 m0189333.ppops.net dellhw_exporter[3383651]: time="2024-03-27T19:58:31Z" level=error msg="storage_vdisk collector failed after 0.082001s: strconv.ParseFloat: parsing \"\": invalid syntax"
Mar 27 19:58:37 m0189333.ppops.net dellhw_exporter[3383651]: time="2024-03-27T19:58:37Z" level=error msg="storage_vdisk collector failed after 0.095158s: strconv.ParseFloat: parsing \"\": invalid syntax"
Mar 27 19:58:46 m0189333.ppops.net dellhw_exporter[3383651]: time="2024-03-27T19:58:46Z" level=error msg="storage_vdisk collector failed after 0.086354s: strconv.ParseFloat: parsing \"\": invalid syntax"
Mar 27 19:58:52 m0189333.ppops.net dellhw_exporter[3383651]: time="2024-03-27T19:58:52Z" level=error msg="storage_vdisk collector failed after 0.075575s: strconv.ParseFloat: parsing \"\": invalid syntax"
# date
Wed Mar 27 20:17:23 UTC 2024

Happy to provide any other information.

galexrt commented 7 months ago

@nexix I have added a workaround, which hopefully shouldn't break other cases (based on the current test cases).

The workaround will be released in v1.13.11.


The more I have to touch the omreport parsing code, the more I think it'll make sense for me to look into rewriting it to use the omreport JSON command output instead of the semicolon-separated text output format...

nexix commented 7 months ago

Thank you! I would agree that json should be a more consistent approach. We use the json output in some other in house tools without issue.