Can you please link to a tutorial/guide on how I'm actually supposed to implement these?

[Jsalas424]

I'm having issues transitioning from smartmon.sh to smartmon.py and would appreciate some guidance.

I have copied smartmon.py into /usr/local/bin and made it an executable file. I have installed the prometheus python client from here: https://github.com/prometheus/client_python. I have added the --collector.textfile.directory=/var/state/prometheus flag to my node exporter and I was previously successfully scraping metrics using the smartmon.sh program but once I switched the smartmon.py I can't get my service file to work correct.

When I was using smartmon.sh, I was running this service file successfully:

[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter
[Service]
Nice=-10
ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py > /var/state/prometheus/smartmon.prom'

# Write nothing except the output file
ProtectSystem=strict
ReadWritePaths=/var/state/prometheus
# Shell needs a temp directory
PrivateTmp=true
ProtectHome=tmpfs

[Install]
WantedBy=multi-user.target

I've tried two other iterations shown below, but ultimately the python script runs one time and then shuts off. What am I doing wrong?

[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter
[Service]
Nice=-10
#ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py > /var/state/prometheus/smartmon.prom'
#ExecStart=/usr/local/bin/smartmon.py
ExecStart=/bin/sh -c '/usr/local/bin/smartmon.py | sponge /var/state/prometheus/smartmon.prom'

[the rest is the same as the above service file]

Here is also my node_exporter service file:

  GNU nano 5.4                          /etc/systemd/system/node_exporter.service                                   
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus

[Install]
WantedBy=multi-user.target

Thank you for the help!

[dswarbrick]

Neither the smartmon.sh nor the smartmon.py scripts are daemons. They are intended to run once, then exit.

You will need a systemd timer to make the service run at regular intervals, and should change Type=simple to Type=oneshot in your service.

You can find some inspiration in the systemd units installed by the Debian prometheus-node-exporter-collectors package: https://salsa.debian.org/go-team/packages/prometheus-node-exporter-collectors/-/tree/debian/sid/debian

[Jsalas424]

@dswarbrick thank you for the help, I did get this running but have run into another issue. After implementing the systemd timer method, my syslog is flooded with error messages of the form "... was collected before with the same name and label values"

TracheServ node_exporter[654339]: ts=2022-06-19T23:51:12.769Z caller=stdlib.go:105 level=error msg="error gathering metrics: 413 error(s) occurred:\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_errors\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 250GB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S59WNJ0N304506K\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"device_model\" value:\"Samsung SSD 850 EVO 250GB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"EMT01B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S21NNSAFC02256H\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 1TB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S5B3NDFNA02148D\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"device_model\" value:\"Samsung SSD 860 EVO 1TB\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"RVT04B6Q\" > label:<name:\"model_family\" value:\"Samsung based SSDs\" > label:<name:\"serial_number\" value:\"S5B3NDFN915923H\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"device_model\" value:\"WDC WD4003FRYZ-01F0DB0\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"01.01H01\" > label:<name:\"model_family\" value:\"Western Digital Gold\" > label:<name:\"serial_number\" value:\"VBGYEPVF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"device_model\" value:\"ST3000NM0033 81Y9799 81Y3865IBM\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"BB59\" > label:<name:\"model_family\" value:\"\" > label:<name:\"serial_number\" value:\"Z1Y0K9N8\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"device_model\" value:\"TOSHIBA MG03ACA100\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"FL1D\" > label:<name:\"model_family\" value:\"Toshiba 3.5\\\" MG03ACAxxx(Y) Enterprise HDD\" > label:<name:\"serial_number\" value:\"44N2KLTFF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_device_info\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"device_model\" value:\"TOSHIBA MG03ACA100\" > label:<name:\"disk\" value:\"0\" > label:<name:\"firmware_version\" value:\"FL1D\" > label:<name:\"model_family\" value:\"Toshiba 3.5\\\" MG03ACAxxx(Y) Enterprise HDD\" > label:<name:\"serial_number\" value:\"3423KG6HF\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:17517 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:27 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:84 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.3290518032e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:34218 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:7206 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:1956 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.33061639514e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:13054 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:18 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:360 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:36 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.05486405539e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:13054 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:17 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:357 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:36 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:1.05485863923e+11 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:4900 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:198 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:47 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:1.20790859e+08 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:2 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:2.860474765e+09 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:59364 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:52 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reported_uncorrect\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"command_timeout\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:2568 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"hardware_ecc_recovered\" > gauge:<value:1.20790859e+08 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:4458 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:78 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:50387 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:78 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:11826 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:39 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:8.9431202034e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_read\" > gauge:<value:5.0053092712e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:6339 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:149 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:59902 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:149 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:11965 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:38 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:0 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:9.751680071e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_raw_value\" { label:<name:\"device\" value:\"/dev/sdh\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_read\" > gauge:<value:5.2769926446e+10 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:96 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:95 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sda\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:93 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:92 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:8 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:67 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdb\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:97 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:76 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:64 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdc\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:97 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"wear_leveling_count\" > gauge:<value:76 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"erase_fail_count_total\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:64 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdd\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"total_lbas_written\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:127 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_event_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sde\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:200 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:81 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:93 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:94 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:33 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reported_uncorrect\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"command_timeout\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"airflow_temperature_cel\" > gauge:<value:52 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"temperature_celsius\" > gauge:<value:48 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"hardware_ecc_recovered\" > gauge:<value:49 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"current_pending_sector\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"offline_uncorrectable\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdf\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"udma_crc_error_count\" > gauge:<value:200 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"raw_read_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_up_time\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"start_stop_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"reallocated_sector_ct\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"seek_error_rate\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_on_hours\" > gauge:<value:1 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"spin_retry_count\" > gauge:<value:101 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"power_cycle_count\" > gauge:<value:100 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > label:<name:\"name\" value:\"load_cycle_count\" > gauge:<value:99 > } was collected before with the same name and label values\n* [from Gatherer #2] collected metric \"smartmon_attr_value\" { label:<name:\"device\" value:\"/dev/sdg\" > label:<name:\"disk\" value:\"0\" > la

My temporary solution was editing the service file to set StandardError=null as such:

  GNU nano 5.4                          /etc/systemd/system/node_exporter.service                                   
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus
StandardError=null

[Install]
WantedBy=multi-user.target

Any suggestions on how to deal with this?

[dswarbrick]

@Jsalas424 This sounds like the collector is running (and appending to the spool file) multiple times within the same scrape interval. Do you perhaps have it still enabled as a cronjob, or perhaps have multiple instances of the systemd timer, executing the service multiple times?

If you see duplicate metrics in the /var/state/prometheus/smartmon.prom, that would be a strong indicator of such a thing.

[Jsalas424]

Hi @dswarbrick, per your advice here's what I found in that directory:

root@TracheServ:~# ls /var/state/prometheus/
smartctl.prom  smartmon.prom

root@TracheServ:~# tail /var/state/prometheus/smartmon.prom
# HELP smartmon_smartctl_run SMART metric smartctl_run
# TYPE smartmon_smartctl_run gauge
smartmon_smartctl_run{device="/dev/sda",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdb",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdc",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdd",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sde",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdf",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdg",disk="0"} 1655020313
smartmon_smartctl_run{device="/dev/sdh",disk="0"} 1655020313

root@TracheServ:~# tail /var/state/prometheus/smartctl.prom
# HELP smartmon_smartctl_run SMART metric smartctl_run
# TYPE smartmon_smartctl_run gauge
smartmon_smartctl_run{device="/dev/sda",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdb",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdc",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdd",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sde",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdf",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdg",disk="0"} 1652064537
smartmon_smartctl_run{device="/dev/sdh",disk="0"} 1652064537

root@TracheServ:~# ls /usr/local/bin/
cmdprox.py  node_exporter  pip  pip3  pip3.7  prox  pve_exporter  smartmon.py  smartmon.sh

root@TracheServ:~# ls /etc/systemd/system
ceph.target.wants                       node_exporter.service            sshd.service
dbus-org.freedesktop.timesync1.service  nut-server.service               sysinit.target.wants
default.target.wants                    prometheus-pve-exporter.service  syslog.service
getty.target.wants                      pve-manager.service              timers.target.wants
ifupdown-wait-online.service            remote-fs.target.wants           zed.service
influxd.service                         smartd.service                   zfs-import.target.wants
iscsi.service                           smart_exporter.service           zfs.target.wants
mnt-pve-spare.mount                     smart_exporter.service.save      zfs-volumes.target.wants
multi-user.target.wants                 smart_exporter.timer
network-online.target.wants             sockets.target.wants

So I just deleted the smartmon.sh, but that didn't fix the issue. So I kept hunting and here's what else I found.

root@TracheServ:~# cat /etc/systemd/system/smart_exporter.timer
[Unit]
Description=Run smart metrics collection every 5 minutes
ConditionPathExists=/usr/sbin/smartctl
ConditionPathExistsGlob=|/dev/sd*
ConditionPathExistsGlob=|/dev/hd*
# TODO: Figure out if NVMe returns usable metrics through smartmon.sh.
#ConditionPathExistsGlob=|/dev/nvme*

[Timer]
OnBootSec=0
OnUnitActiveSec=5min

[Install]
WantedBy=timers.target

root@TracheServ:~# cat /etc/systemd/system/smart_exporter.service 
[Unit]
Description=Export smartctl metrics to Prometheus Node Exporter

[Service]
Type=oneshot
Nice=-10
ExecStart=/bin/sh -c 'exec /usr/local/bin/smartmon.py | sponge /var/state/prometheus/smartmon.prom' 

# Write nothing except the output file
ProtectSystem=strict
ReadWritePaths=/var/state/prometheus
# Shell needs a temp directory
PrivateTmp=true
ProtectHome=tmpfs

[Install]
WantedBy=multi-user.target

root@TracheServ:~# cat /etc/systemd/system/node_exporter.service  
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter --collector.textfile.directory=/var/state/prometheus
StandardError=null
# Add that --collector.textfile flag after execstart once you figure out how to use smart monitoring
#--collector.textfile.directory=/var/state/prometheus

[Install]
WantedBy=multi-user.target

root@TracheServ:~# ls /etc/systemd/system/multi-user.target.wants 
smart_exporter.service  smartmontools.service

So there i a smartmontools/service AND a smart_exporter.service but I can't figure out how to find what's triggering both of these.

Any other hints?

[dswarbrick]

@Jsalas424 You have two metrics files in your /var/state/prometheus directory with identical metrics (albeit different timestamps). This is why you are seeing the error message.

It looks like the smartctl.prom file has the older timestamps (by roughly 34 days), so presumably whatever collector was producing that has stopped. If you remove the smartctl.prom file, node_exporter should stop complaining.

[Jsalas424]

@dswarbrick you are correct, the last step was to delete smartctl.prom! Thanks for everything!