jbuitmann commented 4 months ago

Disk Usage Data Not Displaying

Description

The disk usage data is no longer being received by the Jetson Stats Node Exporter. While other metrics are correctly reported so the rest of the node exporter work as expected, the disk usage field remains empty. This issue persists across multiple versions of the node exporter.

To verify the issue, I tested with jtop (jetson stats), and it correctly displays disk usage information, suggesting the problem is specific to the node exporter.

Expected Behavior

Disk usage data should be displayed in the node exporter, similar to the correct readings seen in jtop.

Actual Behavior

The disk usage field remains empty, with no data being received.

Environment

Jetpack Version [5.1.3]
Jetson Stats Node Exporter Version: 0.0.4 (also tested with 0.0.5 and 0.0.6)

Additional Information

The issue appears to be isolated to the node exporter, as other tools like jtop can correctly read and display disk usage data.
No error messages are being logged related to disk usage in the node exporter logs.

Screenshot of Error

Request for Help

Thank you for your assistance!

laminair commented 2 months ago

Sorry for the delayed response. I just tried to reproduce this issue and did not succeed here. This is the code that reports the disk usage. Please note the mount point definition. Do you have a different root mount point?

    def __disk(self):
        disk_gauge = GaugeMetricFamily(
            name="disk",
            documentation=f"Local Storage Statistics from Jetson Stats (unit: {self.jetson.disk_units})",
            labels=["mountpoint", "statistic"],
            unit="GB"
        )
        for mountpoint, disk_info in self.jetson.disk.items():
            if mountpoint == "/":
                disk_gauge.add_metric(["total"], value=disk_info["total"])
                disk_gauge.add_metric(["used"], value=disk_info["used"])
                disk_gauge.add_metric(["free"], value=disk_info["free"])
                disk_gauge.add_metric(["percent"], value=disk_info["percent"])

        return disk_gauge

If you have the same mount point (which is the default for JetsonOS), can you please check if self.jetson.disk reports anything at all? If you get an empty dictionary, the update does not work properly. I had this in the past when the monitoring frequency was different from the jtop monitoring frequency.

Here is the prometheus export for one of my machines:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 121.0
python_gc_objects_collected_total{generation="1"} 290.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 57.0
python_gc_collections_total{generation="1"} 5.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="8",patchlevel="10",version="3.8.10"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 3.36654336e+08
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.4313856e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.72552092438e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 18.36
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 9.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1024.0
# HELP cpu_Hz CPU Statistics from Jetson Stats (ARMv8 Processor rev 1 (v8l))
# TYPE cpu_Hz gauge
cpu_Hz{core="0",statistic="freq"} 729600.0
cpu_Hz{core="0",statistic="min_freq"} 729600.0
cpu_Hz{core="0",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="0",statistic="val"} 99.0
cpu_Hz{core="1",statistic="freq"} 729600.0
cpu_Hz{core="1",statistic="min_freq"} 729600.0
cpu_Hz{core="1",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="1",statistic="val"} 99.0
cpu_Hz{core="2",statistic="freq"} 729600.0
cpu_Hz{core="2",statistic="min_freq"} 729600.0
cpu_Hz{core="2",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="2",statistic="val"} 95.04950495049505
cpu_Hz{core="3",statistic="freq"} 729600.0
cpu_Hz{core="3",statistic="min_freq"} 729600.0
cpu_Hz{core="3",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="3",statistic="val"} 100.0
cpu_Hz{core="4",statistic="freq"} 729600.0
cpu_Hz{core="4",statistic="min_freq"} 729600.0
cpu_Hz{core="4",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="4",statistic="val"} 98.01980198019803
cpu_Hz{core="5",statistic="freq"} 729600.0
cpu_Hz{core="5",statistic="min_freq"} 729600.0
cpu_Hz{core="5",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="5",statistic="val"} 99.0
cpu_Hz{core="6",statistic="freq"} 729600.0
cpu_Hz{core="6",statistic="min_freq"} 729600.0
cpu_Hz{core="6",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="6",statistic="val"} 100.0
cpu_Hz{core="7",statistic="freq"} 729600.0
cpu_Hz{core="7",statistic="min_freq"} 729600.0
cpu_Hz{core="7",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="7",statistic="val"} 100.0
cpu_Hz{core="8",statistic="freq"} 729600.0
cpu_Hz{core="8",statistic="min_freq"} 729600.0
cpu_Hz{core="8",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="8",statistic="val"} 100.0
cpu_Hz{core="9",statistic="freq"} 729600.0
cpu_Hz{core="9",statistic="min_freq"} 729600.0
cpu_Hz{core="9",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="9",statistic="val"} 100.0
cpu_Hz{core="10",statistic="freq"} 729600.0
cpu_Hz{core="10",statistic="min_freq"} 729600.0
cpu_Hz{core="10",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="10",statistic="val"} 99.0
cpu_Hz{core="11",statistic="freq"} 729600.0
cpu_Hz{core="11",statistic="min_freq"} 729600.0
cpu_Hz{core="11",statistic="max_freq"} 2.2016e+06
cpu_Hz{core="11",statistic="val"} 100.0
# HELP gpu_utilization_percentage_Hz GPU Statistics from Jetson Stats
# TYPE gpu_utilization_percentage_Hz gauge
gpu_utilization_percentage_Hz{nvidia_gpu="freq",statistic="ga10b"} 306000.0
gpu_utilization_percentage_Hz{nvidia_gpu="min_freq",statistic="ga10b"} 306000.0
gpu_utilization_percentage_Hz{nvidia_gpu="max_freq",statistic="ga10b"} 1.3005e+06
# HELP ram_kB Memory Statistics from Jetson Stats (unit: kB)
# TYPE ram_kB gauge
ram_kB{statistic="total"} 6.4287204e+07
ram_kB{statistic="used"} 1.188576e+06
ram_kB{statistic="buffers"} 95936.0
ram_kB{statistic="cached"} 825292.0
ram_kB{statistic="lfb"} 15158.0
ram_kB{statistic="free"} 6.226566e+07
# HELP gpuram_kB Video Memory Statistics from Jetson Stats
# TYPE gpuram_kB gauge
gpuram_kB{nvidia_gpu="mem",statistic="ga10b"} 0.0
# HELP swap_kB Swap Statistics from Jetson Stats
# TYPE swap_kB gauge
swap_kB{statistic="total"} 3.2143584e+07
swap_kB{statistic="used"} 0.0
swap_kB{statistic="cached"} 0.0
# HELP emc_Hz EMC Statistics from Jetson Stats
# TYPE emc_Hz gauge
emc_Hz{statistic="total"} 2.133e+06
emc_Hz{statistic="used"} 3.199e+06
emc_Hz{statistic="cached"} 204000.0
# HELP temperature_C Temperature Statistics from Jetson Stats (unit: °C)
# TYPE temperature_C gauge
temperature_C{statistic="CPU"} 34.625
temperature_C{statistic="CV0"} -256.0
temperature_C{statistic="CV1"} -256.0
temperature_C{statistic="CV2"} -256.0
temperature_C{statistic="GPU"} -256.0
temperature_C{statistic="SOC0"} 32.187
temperature_C{statistic="SOC1"} 31.375
temperature_C{statistic="SOC2"} 31.656
temperature_C{statistic="Tboard"} 22.0
temperature_C{statistic="Tdiode"} 23.5
temperature_C{statistic="tj"} 34.343
# HELP integrated_power Power Statistics from internal power sensors (unit: mW/V/A)
# TYPE integrated_power gauge
integrated_power{statistic="voltage"} 5000.0
integrated_power{statistic="current"} 0.0
integrated_power{statistic="critical"} 3000.0
integrated_power{statistic="power"} 0.0
integrated_power{statistic="avg_power"} 0.0
integrated_power{statistic="voltage"} 0.0
integrated_power{statistic="current"} 0.0
integrated_power{statistic="critical"} 0.0
integrated_power{statistic="power"} 0.0
integrated_power{statistic="avg_power"} 0.0
integrated_power{statistic="voltage"} 5048.0
integrated_power{statistic="current"} 60.0
integrated_power{statistic="critical"} 81900.0
integrated_power{statistic="power"} 302.0
integrated_power{statistic="avg_power"} 302.0
integrated_power{statistic="voltage"} 20064.0
integrated_power{statistic="current"} 20.0
integrated_power{statistic="critical"} 81900.0
integrated_power{statistic="power"} 401.0
integrated_power{statistic="avg_power"} 401.0
integrated_power{statistic="voltage"} 20064.0
integrated_power{statistic="current"} 120.0
integrated_power{statistic="critical"} 81900.0
integrated_power{statistic="power"} 2407.0
integrated_power{statistic="avg_power"} 2406.0
integrated_power{statistic="voltage"} 5048.0
integrated_power{statistic="current"} 700.0
integrated_power{statistic="critical"} 81900.0
integrated_power{statistic="power"} 3533.0
integrated_power{statistic="avg_power"} 3533.0
# HELP integrated_power_mW Power Statistics from internal power sensors (unit: mW)
# TYPE integrated_power_mW gauge
integrated_power_mW{statistic="power"} 6643.0
integrated_power_mW{statistic="avg_power"} 6642.0
# HELP disk_GB Local Storage Statistics from Jetson Stats (unit: GB)
# TYPE disk_GB gauge
disk_GB{mountpoint="total"} 981.814759424
disk_GB{mountpoint="used"} 37.880795136
disk_GB{mountpoint="free"} 893.985026048
disk_GB{mountpoint="percent"} 4.0999999999999995e-09
# HELP uptime_s Machine Uptime Statistics from Jetson Stats
# TYPE uptime_s gauge
uptime_s{statistic="alive"} 64494.26

jbuitmann commented 2 weeks ago

Firstly, thank you very much for trying to help me! :) Sorry for the late response. We've been experiencing some issues with our project, particularly with the Jetson. It "died" a few times, and we still don't know why. I also wanted to see if updating everything might solve the issue, including jetson_stats_node_exporter (0.1.0), jtop (4.2.9), and most importantly JetPack to version 6.0.

You suggested checking the mountpoint, and from what I can tell, it seems to be fine.

I'm not sure if the screenshot is clear enough, but it shows jtop displaying the disk space correctly (120/455GB).

What's still strange is that I hadn't updated jetson_exporter, jtop, or JetPack when it stopped working. I'm the only one using the device, and one day the data was just missing, even though the rest of your script continues to run perfectly fine.

laminair / jetson_stats_node_exporter

No disk usage data exported anymore #7