Problem parsing VDO fields: online

Garagoth commented 1 year ago

lvm exporter has trouble parsing some values, and reports 4 errors (but logs just two):

Jun 26 08:31:55 xxx prometheus-lvm-exporter[19726]: 2023/06/26 08:31:55 Scrape failed: 4 error(s) occurred:
Jun 26 08:31:55 xxx prometheus-lvm-exporter[19726]: * field vdo_compression_state: strconv.ParseFloat: parsing "online": invalid syntax
Jun 26 08:31:55 xxx prometheus-lvm-exporter[19726]: * field vdo_index_state: strconv.ParseFloat: parsing "online": invalid syntax
Jun 26 08:32:08 xxx prometheus-lvm-exporter[19726]: 2023/06/26 08:32:08 Scrape failed: 4 error(s) occurred:
Jun 26 08:32:08 xxx prometheus-lvm-exporter[19726]: * field vdo_compression_state: strconv.ParseFloat: parsing "online": invalid syntax
Jun 26 08:32:08 xxx prometheus-lvm-exporter[19726]: * field vdo_index_state: strconv.ParseFloat: parsing "online": invalid syntax

hansmi commented 1 year ago

Looks like the test data doesn't contain all VDO fields, so the vdo_compression_state and vdo_index_state fields weren't fully exercised. They're always strings (relevant source code in lib/metadata/vdo_manip.c).

@Garagoth, I'm happy to submit a fix similar to commit c17044b to treat the VDO fields as info instead of metrics. Would you be able to provide me with a raw JSON snippet with real values for use in tests? The command would be as follows (apply sudo, etc. as necessary):

/usr/sbin/lvm lvs --all --config 'global/suffix=false global/units=b report/binary_values_as_numeric=1 report/output_format=json report/buffered=false report/time_format=%s' \
  --configreport lv --options \
  lv_uuid,lv_name,vdo_compression_state,vdo_index_state,vdo_operating_mode,vdo_saving_percent,vdo_used_size

and reports 4 errors (but logs just two)

That's interesting to say the least. prometheus.MultiError‎ uses len() of a slice just before iterating over the same. I can't reproduce your output with simulated data triggering errors.

This might be far-fetched, but is it possible that your logging imposes some sort of rate limitation which would drop two additional lines per scrape failure?

Garagoth commented 1 year ago

  {
      "report": [
          {
              "lv": [
                  {"lv_uuid":"dylifl-81AF-n2mG-kFKo-Ntxe-mnNQ-3eB7fF", "lv_name":"vpool0", "vdo_compression_state":"online", "vdo_index_state":"online", "vdo_operating_mode":"normal", "vdo_saving_percent":"55.37", "vdo_used_size":"2788999639040"},
                  {"lv_uuid":"r7Btja-nL3P-Co38-X2K4-DLWS-LMhT-LajcQa", "lv_name":"lv_db_backup", "vdo_compression_state":"online", "vdo_index_state":"online", "vdo_operating_mode":"normal", "vdo_saving_percent":"55.37", "vdo_used_size":"2788999639040"},
                  {"lv_uuid":"tHnHR1-bjg8-ycJX-eyhR-heEp-L4qs-S510dO", "lv_name":"[vpool0_vdata]", "vdo_compression_state":"", "vdo_index_state":"", "vdo_operating_mode":"", "vdo_saving_percent":"", "vdo_used_size":""},
                  {"lv_uuid":"5UyOrx-zG6u-w7cF-G2Ny-HmVd-0lsb-4J6lG2", "lv_name":"lv_data", "vdo_compression_state":"", "vdo_index_state":"", "vdo_operating_mode":"", "vdo_saving_percent":"", "vdo_used_size":""},
                  {"lv_uuid":"lGs7Tn-1Fyv-jMHN-aniN-r8qk-duEI-B9Xl9e", "lv_name":"swap", "vdo_compression_state":"", "vdo_index_state":"", "vdo_operating_mode":"", "vdo_saving_percent":"", "vdo_used_size":""},
                  {"lv_uuid":"cqZnVQ-FUPG-btWz-3fff-cMX5-wViK-8MM3QY", "lv_name":"root", "vdo_compression_state":"", "vdo_index_state":"", "vdo_operating_mode":"", "vdo_saving_percent":"", "vdo_used_size":""}
              ]
          }
      ]
  }

My journal logs message that some messages were suppressed when it does that, but nothing here. I pasted above two error logs that are consecutive, nothing in between.

hansmi commented 1 year ago

I pushed a commit converting vdo_compression_state and vdo_index_state to info fields. Do you want me to tag a new release?

With your test data the error as of commit 539a714 (HEAD before the aforementioned change) written by the program is as follows:

4 error(s) occurred:
* field vdo_compression_state: strconv.ParseFloat: parsing "online": invalid syntax
* field vdo_compression_state: strconv.ParseFloat: parsing "online": invalid syntax
* field vdo_index_state: strconv.ParseFloat: parsing "online": invalid syntax
* field vdo_index_state: strconv.ParseFloat: parsing "online": invalid syntax

It's possible that duplicate lines are suppressed by logging. I don't have any other explanation for the mismatch.

hansmi commented 1 year ago

I've tagged version 0.3.3.

Garagoth commented 1 year ago

Thanks! Installed, seems to be working fine!

hansmi / prometheus-lvm-exporter

Problem parsing VDO fields: online #29