Readings are delayed on latest version

[Franscobec]

Hi, I've noticed that after upgrading the led_matrix_monitoring and led_matrix_daemon from Archlinux's AUR, the readings of the LED Matrix are delayed by around 10 seconds, which means the readings are inaccurate.

I am verifying this using stress-ng -c32 and watching btop's data against what is displayed on the LEDs.

Here is my config:

socket = "/var/run/led-matrix/led-matrix.sock"

[collector]
max_history_samples = 10
sample_interval = "100ms"

disk_names = [
    { Name = { Equal = "nvme0n1" } },
]

network_interfaces = [
    { Name = { Equal = "wlp5s0" } },
]

temperatures = [
    { StartsWith = "k10temp" },
]

[render]
left = [
    { Battery = { start_y = 0, max_height = 34 } }
]

right = [
    { Cpu = { mid_point = 10, max_height = 10, k = 1.0 } },
    { AverageCpu = { start_x = 7, start_y = 20, end_y = 9, k = 1.0 } },
    { Temp = { max_value = 100, start_y = 33, start_x = 0, end_x = 9, k = 3.0 } },
    { Temp = { max_value = 100, start_y = 32, start_x = 0, end_x = 9, k = 3.0 } },
    { Temp = { max_value = 100, start_y = 31, start_x = 0, end_x = 9, k = 3.0 } },
    { Temp = { max_value = 100, start_y = 30, start_x = 0, end_x = 9, k = 3.0 } },
    { Network = { mid_point = 25, max_height = 4, k = 6.0 } }
]

daemon.toml has not been edited from the default config. I am using Archlinux with systemd and the Linux-Zen kernel

I had the same config on the previous version without issues, and reverting to the old version fixes the issue but brings back the previously mentioned issue about LEDs still being illuminated even when they're not supposed to.

[night-crawler]

Hi @Franscobec,

Thanks for the report. Classically, it does not reproduce on my machine. Also, I fail to see what could affect such behavior in this change.

Nevertheless, let's dig it through if you don't mind. The daemon returns the length of its current queue after adding a new item to it. By default queue is limited to 100 samples. If it takes long to render a sample, it can explain why it's happening on your machine. To confirm that, could you please checkout the daemon repo and:

git clone https://github.com/night-crawler/led_matrix_daemon
cd led_matrix_daemon/test_data
./curl_test_b64.sh

You will see something like

{"queue_len":0,"queued":true}

If queue_len != 0 it means that the we found the culprit. Yet another way to verify the lag is:

sudo systemctl stop --now led_matrix_monitoring.service 

If you still see the data rendering after stopping the monitoring service, it's the same.

If it confirms either way, you can mitigate the problem by reducing the queue length to 1 in /etc/led_matrix/daemon.toml. After the queue is filled, the daemon will start blocking calls, so it will start blocking the producer and you will have correct data but with lower FPS. Identically, you can play with sample_interval = "170ms" in /etc/led_matrix/monitoring.toml and find a value that works for you.

In the meantime I'll add debug logs with timings and take a look at what versions might have changed that could affect rendering time.

[Franscobec]

Hi, thanks for the response.

My queue_len was at 100, which explains it. With the changes that you mention, I also fail to see why my config would suddenly cause this. I've found that setting it back to 170ms fixes the issue and setting it to any lower causes the issue (yes, even 169ms).

I've also got a new issue while testing things, which included disconnecting and reconnecting the trackpad (which turns off the input deck entirely) the daemon is unable to restart properly or restore communication with the LED Matrix. Anyway maybe this is a limitation of the platform and I'll probably open a new issue for this if necessary.

Setting the queue length to 1 in the daemon also fixed the issue, but it was less nice to look at so I let the limit as-is and set the monitoring to 170ms instead.

[night-crawler]

Hi,

When developing I had a thought or two about it (what to do if there's a bad connection, should we have timeouts for rendering, should we fail fast or try stay alive and do retrying under the hood and so on). For the initial stage I decided to play it stupid and do nothing.

If it hurts (your) UX then I think it's a bug, so feel free to open another one.