lvm_up metric error

[SckyzO]

Hello,

Thank you for your work, lvm_exporter work fine.

Just I have an error with some servers with lvm_up metric :

# HELP lvm_up Whether scrape was successful
# TYPE lvm_up gauge
lvm_up{status="5 error(s) occurred:\n* field copy_percent: strconv.ParseFloat: parsing \"0,00\": invalid syntax\n* field data_percent: strconv.ParseFloat: parsing \"12,33\": invalid syntax\n* field metadata_percent: strconv.ParseFloat: parsing \"27,67\": invalid syntax\n* field snap_percent: strconv.ParseFloat: parsing \"12,33\": invalid syntax\n* field sync_percent: strconv.ParseFloat: parsing \"0,00\": invalid syntax"} 0

[root@server prometheus-lvm-exporter_0.3.2_linux_amd64]# ./prometheus-lvm-exporter
2022/12/07 16:22:12 LVM command: /usr/sbin/lvm fullreport --all --config 'global/suffix=false global/units=b report/binary_values_as_numeric=1 report/output_format=json report/buffered=false report/time_format=%s' --configreport pv --options pv_uuid,pv_name,pv_all --configreport lv --options lv_uuid,lv_name,lv_all --configreport vg --options vg_uuid,vg_name,vg_all --configreport seg --options -all --configreport pvseg --options -all
2022/12/07 16:22:12 level=info msg="Listening on" address=[::]:9845
2022/12/07 16:22:12 level=info msg="TLS is disabled." http2=false address=[::]:9845
WARNING: Ignoring duplicate config value: system_id_source
  WARNING: Ignoring duplicate config value: system_id_source
  WARNING: Ignoring duplicate config value: system_id_source
2022/12/07 16:22:18 Scrape failed: 5 error(s) occurred:
* field copy_percent: strconv.ParseFloat: parsing "0,00": invalid syntax
* field data_percent: strconv.ParseFloat: parsing "12,33": invalid syntax
* field metadata_percent: strconv.ParseFloat: parsing "27,67": invalid syntax
* field snap_percent: strconv.ParseFloat: parsing "12,33": invalid syntax
* field sync_percent: strconv.ParseFloat: parsing "0,00": invalid syntax
...

Do you need more informations ?

Regards,

[hansmi]

@SckyzO, there's a good chance that you have a locale configuration using a comma (,) as a decimal separator. We can fix that by overriding the LC_ALL environment variable when invoking the LVM commands. However, I'd like to first confirm this as the cause. Can you please comment with the output of the locale command (likely /usr/bin/locale)?

[hansmi]

@SckyzO, I'm tried to reproduce the problem by forcefully using a non-English locale in prometheus-lvm-exporter --command='sudo env LC_ALL=fr_FR.utf8 lvm'. Parsing was successful for copy_percent, etc. I ran the tests on Debian Bullseye with LVM 2.03.11-2.1.

As a next step I invoked lvs directly. LC_ALL=fr_FR.utf8 lvs shows locale-formatted values such as 90,00g (vs. 90.00g for C and en_US.utf-8). Human-friendly formatting is disabled in the JSON output requested by the exporter (--config '[…] global/units=b […]'): "lv_size":"96636764160". global/units is mentioned in the changelog for LVM 2.02.22 from February 2007, making it an unlikely cause.

Which version of LVM are you using on what Linux distribution?

[SckyzO]

Hello @hansmi

You were right. My problem was the locale send when my computer connect to my servers with SSH. Wy Windows is French language, my WSL Ubuntu have locale in US, but Windows send special env to WSL ... So just, I commented "SendEnv LANG LC_*" in /etc/ssh/ssh_config, reload my ssh connection, and retry. It working fine ! :)

Thank you for your help, but it was a PEBCAK 😊😂

And, for information my system was :

[root@server1 ~] # lvs --version
WARNING: Ignoring duplicate config value: system_id_source
  LVM version:     2.03.14(2)-RHEL8 (2021-10-20)
  Library version: 1.02.181-RHEL8 (2021-10-20)
  Driver version:  4.43.0
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service --enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with-vdo=internal --with-vdo-format=/usr/bin/vdoformat --with-integrity=internal --disable-silent-rules

locale :

[root@server1 ~]# locale
LANG=fr_FR.UTF-8
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

[root@server1 ~]# lsb_release -a
LSB Version:    :core-4.1-amd64:core-4.1-noarch
Distributor ID: RedHatEnterprise
Description:    Red Hat Enterprise Linux release 8.6 (Ootpa)
Release:        8.6
Codename:       Ootpa

[hansmi]

@SckyzO, what surprises me is that the locale has any influence at all. LVM's JSON output is supposed to be locale-independent. We may be dealing with a bug and/or a misunderstanding. Disabling environment variables in SSH merely masks the problem.

If you don't mind I'd like to investigate further.

Can you run the LVM command logged by the exporter under the original environment, i.e. with SSH sending LANG and LC_*? The command as copied from the first comment in this thread:

/usr/sbin/lvm fullreport --all --config 'global/suffix=false global/units=b report/binary_values_as_numeric=1 report/output_format=json report/buffered=false report/time_format=%s' --configreport pv --options pv_uuid,pv_name,pv_all --configreport lv --options lv_uuid,lv_name,lv_all --configreport vg --options vg_uuid,vg_name,vg_all --configreport seg --options -all --configreport pvseg --options -all

Obviously you can also set the environment for the command using LANG=fr_FR.UTF-8 LC_ALL=fr_FR.UTF-8 /usr/sbin/lvm fullreport […] instead of making SSH configuration changes.

Is the output for copy_percent, data_percent, etc. formatted with commas (,) or periods (.)?

[hansmi]

Closing again. I can't investigate further without more information.