I got this issue with running prometheus_plugin and fix it.
Configured stat set:
Clusters: [isilon01]
Update Interval: 300
Stat Keys: {'ifs.bytes.free', 'ifs.bytes.avail', 'ifs.percent.avail', 'ifs.bytes.total', 'ifs.percent.used', 'ifs.bytes.used', 'ifs.percent.free'}
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run
File "/isilon/isi_data_insights_daemon.py", line 702, in _query_and_process_stats1
self._process_stats_func(cluster.name, results, derived_stats_processors)
File "/isilon/isi_data_insights_daemon.py", line 746, in _process_stats_with_derived_stats
self._stats_processor.process_stat(cluster_name, stat)
File "/isilon/prometheus_plugin.py", line 76, in process_stat
_process_list(tags, stat.key, stat.value)
File "/isilon/prometheus_plugin.py", line 88, in _process_list
_process_dict(tags, basekey, elem)
File "/isilon/prometheus_plugin.py", line 97, in _process_dict
for k in statdict.keys():
RuntimeError: dictionary changed size during iteration
2021-12-27T14:37:38Z <Greenlet at 0x7fb77e858260: <bound method IsiDataInsightsDaemon._query_and_process_stats1 of <isi_data_insights_daemon.IsiDataInsightsDaemon object at 0x7fb781767a00>>(isilon01, {'node.clientstats.active.siq', 'cluster.protostat, [], [], [], [], False)> failed with RuntimeError
Also my PR contains requirements.txt list changes: added prometheus_client
I got this issue with running prometheus_plugin and fix it.
Also my PR contains requirements.txt list changes: added prometheus_client