search

threefoldtech / 0-stor_v2

Apache License 2.0
3 stars 1 forks source link

New backends added with SIGUSR1 aren't included with Prometheus metrics #120

Open scottyeager opened 3 months ago

scottyeager commented 3 months ago

I updated the config file of a zstor process running in monitor mode and reloaded the config by issuing SIGUSR1. The new backends now show up when I run the status command, but they have not been added to the Prometheus metrics.

Compare:

# zstor -c /etc/zstor-default.toml status
+---------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| backend                                                             | reachable | objects | used space | free space | usage percentage |
+=====================================================================+===========+=========+============+============+==================+
| [44a:5fd6:b716:d147:f5ae:7088:a381:4976]:9900 - 81-42998-node1meta0 | No        |       - |          - |          - |                - |
+---------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4b8:280d:e777:5252:f985:19cc:dcfd:7624]:9900 - 81-42999-node3meta0 | Yes       |      93 |      44208 | 1073741824 |                0 |
+---------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4d0:6235:9b0e:328b:b0f8:4c1c:5ed8:b673]:9900 - 81-43000-node5meta0 | Yes       |      93 |      44208 | 1073741824 |                0 |
+---------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [564:7d94:d5cc:f37b:4783:35d6:6b37:a40]:9900 - 81-43001-node7meta0  | Yes       |      93 |      44208 | 1073741824 |                0 |
+---------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| backend                                                                    | reachable | objects | used space | free space | usage percentage |
+============================================================================+===========+=========+============+============+==================+
| [44a:5fd6:b716:d147:f5ae:7088:a381:4976]:9900 - 81-43002-node1backend0     | No        |       - |          - |          - |                - |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4b8:280d:e777:5252:f985:19cc:dcfd:7624]:9900 - 81-43003-node3backend0     | Yes       |     620 | 1071916880 | 1073741824 |               99 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4d0:6235:9b0e:328b:b0f8:4c1c:5ed8:b673]:9900 - 81-43004-node5backend0     | Yes       |     620 | 1071916880 | 1073741824 |               99 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [564:7d94:d5cc:f37b:4783:35d6:6b37:a40]:9900 - 81-43005-node7backend0      | Yes       |     620 | 1071916880 | 1073741824 |               99 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [44a:5fd6:b716:d147:f5ae:7088:a381:4976]:9900 - 81-43148-node1backendsnew0 | No        |       - |          - |          - |                - |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4b8:280d:e777:5252:f985:19cc:dcfd:7624]:9900 - 81-43149-node3backendsnew0 | Yes       |       0 |          0 | 1073741824 |                0 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [4d0:6235:9b0e:328b:b0f8:4c1c:5ed8:b673]:9900 - 81-43150-node5backendsnew0 | Yes       |       0 |          0 | 1073741824 |                0 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
| [564:7d94:d5cc:f37b:4783:35d6:6b37:a40]:9900 - 81-43151-node7backendsnew0  | Yes       |       0 |          0 | 1073741824 |                0 |
+----------------------------------------------------------------------------+-----------+---------+------------+------------+------------------+
# TYPE data_disk_freespace_bytes gauge
data_disk_freespace_bytes{address="[44a:5fd6:b716:d147:f5ae:7088:a381:4976]:9900",backend_type="data",namespace="81-43002-node1backend0"} 996966887424
data_disk_freespace_bytes{address="[4b8:280d:e777:5252:f985:19cc:dcfd:7624]:9900",backend_type="data",namespace="81-43003-node3backend0"} 2997354729472
data_disk_freespace_bytes{address="[4b8:280d:e777:5252:f985:19cc:dcfd:7624]:9900",backend_type="meta",namespace="81-42999-node3meta0"} 2997354729472
data_disk_freespace_bytes{address="[4d0:6235:9b0e:328b:b0f8:4c1c:5ed8:b673]:9900",backend_type="data",namespace="81-43004-node5backend0"} 2997354999808
data_disk_freespace_bytes{address="[4d0:6235:9b0e:328b:b0f8:4c1c:5ed8:b673]:9900",backend_type="meta",namespace="81-43000-node5meta0"} 2997354999808
data_disk_freespace_bytes{address="[564:7d94:d5cc:f37b:4783:35d6:6b37:a40]:9900",backend_type="data",namespace="81-43005-node7backend0"} 2997354733568
data_disk_freespace_bytes{address="[564:7d94:d5cc:f37b:4783:35d6:6b37:a40]:9900",backend_type="meta",namespace="81-43001-node7meta0"} 2997354733568
scottyeager commented 4 weeks ago

Further observation here is that I realized when calling status, backends are shown in the output if and only if they are present in the config file. There's no relation to which backends zstor actually has in its live config.

I was able to confirm that after a SIGUSR1 zstor is using the new backends from the config file, based on the fact that it's uploading data to them. However, the behavior of the new backends not showing up on the metrics endpoint in reproduced in my test today too.