[BUG] Salt Exception with message "Event iteration failed with exception: 'list' object has no attribute 'items'"

[Laurens-IVG]

Description We did not change any configuration, so the setup was running before. We want to deploy a Mailserver using Terraform which is calling the salt stack.

When running a High State, we get the following errors:

Minion Log:

module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): [DEBUG   ] LazyLoaded highstate.output
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): local:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec):     Data failed to compile:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec):     Pillar failed to render with the following messages:
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec):     Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec): ----------
module.product_mailservers[3].module.nutanixvm.null_resource.salt-bootstrap (remote-exec):     Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.

Master Log:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 896, in render_pstate
    **defaults
  File "/usr/lib/python3/dist-packages/salt/template.py", line 99, in compile_template
    ret = render(input_data, saltenv, sls, **render_kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/renderers/jinja.py", line 75, in render
    **kws
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3/dist-packages/salt/utils/templates.py", line 509, in render_jinja_tmpl
    raise SaltRenderError("Jinja variable {}{}".format(exc, out), buf=tmplstr)
salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'addresses'
2023-06-27 15:16:45,724 [salt.pillar      :1224][CRITICAL][134604] Pillar render error: Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
2023-06-27 15:16:45,724 [salt.pillar      :1224][CRITICAL][134604] Pillar render error: Rendering SLS 'mailserver.opendkim' failed. Please see master log for details.
2023-06-27 15:16:45,883 [salt.utils.event :908 ][ERROR   ][134459] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:16:55,288 [salt.utils.event :908 ][ERROR   ][134475] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:18:18,594 [salt.utils.event :908 ][ERROR   ][134515] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:23:25,128 [salt.utils.event :908 ][ERROR   ][134491] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:24:07,291 [salt.utils.event :908 ][ERROR   ][134514] Event iteration failed with exception: 'list' object has no attribute 'items'
2023-06-27 15:27:16,587 [salt.utils.event :908 ][ERROR   ][134521] Event iteration failed with exception: 'list' object has no attribute 'items'

Setup

• [ X] on-prem machine
• [ X] VM - Nutanix
• [ X] in combination with Terraform - TF calls the salt stack do deploy an application

The following file is called when the problem occurs:

opendkim.sls:

{%- if pillar.get('opendkim-key') %}
include:
  - base: opendkim

opendkim-schedule:
  schedule.present:
    - function: state.sls
    - job_args:
      - mailserver.opendkim
    - seconds: 21600
    - splay:
        start: 300
        end: 1500
    - run_on_start: false
    - maxrunning: 1
    - jid_include: True
{%- else %}
opendkim-schedule:
  schedule.absent:
    - name: opendkim-schedule
{%- endif %}

Steps to Reproduce the behavior Run the state either manually or use Terraform.

Expected behavior Salt Stack should be called, running and deploy the Mailserver as configured.

Screenshots I don't think they're needed here.

Versions Report

root@salt01: /var/log/salt # salt --versions-report Salt Version: Salt: 3004.2 Dependency Versions: cffi: 1.15.1 cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.5 gitpython: 2.1.11 Jinja2: 2.10 libgit2: 1.5.0 M2Crypto: Not Installed Mako: 1.1.3 msgpack: 0.5.6 msgpack-pure: Not Installed mysql-python: 1.3.10 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: 1.10.0 Python: 3.7.3 (default, Jan 22 2021, 20:04:44) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.1.2 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.1 System Versions: dist: debian 10 buster locale: UTF-8 machine: x86_64 release: 4.19.0-21-amd64 system: Linux version: Debian GNU/Linux 10 buster

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[s0undt3ch]

What is trying to access that addresses key?

[dmurphy18]

May have similar, see Salt Community Slack , general channel https://saltstackcommunity.slack.com/archives/C7K04SEJC/p1705011335140729

here it is in case contributor not on Slack

kmm
  3:15 PM
In 3006.5 (running on the salt master and minions), I'm running into many occurrences of the error [salt.utils.event :912 ][ERROR   ][2758413] Event iteration failed with exception: 'list' object has no attribute 'items' in the salt master's log file. These errors are immediately preceded by an event in the form:
[salt.utils.event :823 ][DEBUG   ][2758413] Sending event: tag = salt/job/[jid]/ret/[name]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1', 'arg2'], 'schedule': '[scheduler state]', 'jid': '[jid]', 'pid': 656, 'return': ['The function "state.sls" is running as PID 4252 and was started at 2024, Jan 11 22:09:45.272142 with jid [jid]'], 'retcode': 1, 'success': True, '_stamp': '2024-01-11T22:09:49.164465', 'out': 'highstate', 'arg': ['arg1', 'arg2'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}
However, in most other "Sending event:" log messages, the return key has a dictionary value, rather than a list/str (which is what's causing the error). Has anyone previously encountered this before?

[morenski]

@dmurphy18 I'll provide a little more context here about the issue. In one instance, the "Event iteration failed..." was generated as a result of this error in a minion:

2024-01-11 14:35:53,434 [salt.utils.schedule:913 ][ERROR   ][2592540] Unhandled exception running state.sls
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/schedule.py", line 880, in handle_func
    ret["return"] = self.functions[func](*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1357, in sls
    conflict = _check_queue(queue, kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 430, in _check_queue
    conflict = running(concurrent=kwargs.get("concurrent", False))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 381, in running
    active = __salt__["saltutil.is_running"]("state.*")
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/context.py", line 78, in __getitem__
    return self.value()[item]
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 349, in __getitem__
    super().__getitem__(item)  # try to get the item from the dictionary
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/lazy.py", line 105, in __getitem__
    raise KeyError(key)
KeyError: 'saltutil.is_running'

That error in the minion resulted in the following error in the salt master's log file:

2024-01-11 14:32:53,449 [salt.utils.event :823 ][DEBUG   ][2758412] Sending event: tag = salt/job/[jid]/ret/[minion_id]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1'], 'schedule': 'schedule1', 'jid': '[jid]', 'pid': 2592426, 'return': 'Unhandled exception running state.sls', 'success': False, 'retcode': 254, '_stamp': '2024-01-11T22:32:53.449264', 'out': 'highstate', 'arg': ['arg1'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}
2024-01-11 14:32:53,450 [salt.utils.event :912 ][ERROR   ][2758412] Event iteration failed with exception: 'str' object has no attribute 'items'

In another instance, the log message in the salt-master was:

[salt.utils.event :823 ][DEBUG   ][2758403] Sending event: tag = salt/job/[jid]/ret/[minion_id]; data = {'cmd': '_return', 'id': '[minion_id]', 'fun': 'state.sls', 'fun_args': ['arg1'], 'schedule': 'schedule2', 'jid': '[jid]', 'pid': 2593155, 'return': ["Detected conflicting IDs, SLS IDs need to be globally unique.\n    The conflicting ID is 'handle_unsupported_system' and is found in SLS 'base:[salt state1]' and SLS 'base:[salt state2]'"], 'retcode': 1, 'success': True, '_stamp': '2024-01-11T22:42:35.930293', 'out': 'highstate', 'arg': ['arg1'], 'tgt_type': 'glob', 'tgt': '[minion_id]'}

However, for the vast majority of minions, the "Sending event" log message is accompanied by a data dictionary containing a return key with a dictionary as its value. Even more perplexing, it appears restarting the affected minions resolves the issue, but only until the next time the salt master becomes overloaded & causes minion timeouts and associated errors. It might be worth mentioning that these timeouts are frequently associated with the following error:

2024-01-11 22:31:48,936 [salt.utils.schedule:913 ][ERROR   ][1779876] Unhandled exception running state.sls
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1157, in _channel_send
    return self.channel.send(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 341, in send
    ret = yield self._crypted_transfer(load, timeout=timeout, raw=raw)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 294, in _crypted_transfer
    ret = yield _do_transfer()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", line 275, in _do_transfer
    data = yield self.transport.send(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 909, in send
    ret = yield self.message_client.send(load, timeout=timeout)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1064, in run
    yielded = self.gen.throw(*exc_info)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", line 589, in send
    recv = yield future
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/gen.py", line 1056, in run
    value = future.result()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
salt.exceptions.SaltReqTimeoutError: Message timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/schedule.py", line 880, in handle_func
    ret["return"] = self.functions[func](*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/modules/state.py", line 1466, in sls
    high_, errors = st_.render_highstate({opts["saltenv"]: mods})
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 4642, in render_highstate
    statefiles = fnmatch.filter(self.avail[saltenv], sls_match)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/state.py", line 3734, in __getitem__
    self._avail[saltenv] = self._hs.client.list_states(saltenv)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 385, in list_states
    for path in self.file_list(saltenv):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1376, in file_list
    return self._channel_send(
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/fileclient.py", line 1162, in _channel_send
    raise SaltClientError(
salt.exceptions.SaltClientError: File client timed out after 1691084572

Master version info:

[me@master]# salt-run --versions-report
Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: rocky 8.9 Green Obsidian
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.9.1.el8_9.x86_64
        system: Linux
       version: Rocky Linux 8.9 Green Obsidian

Minion version info:

[me@minion]# salt-call --versions-report
Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.15.0-1038-gcp
        system: Linux
       version: Ubuntu 20.04.6 focal

[lkubb]

I'm also seeing this message, but as mentioned it's only happening as a side effect when a state run fails otherwise. I'm reasonably sure it's caused by:

https://github.com/saltstack/salt/blob/daef7613bc359822a4f04ffdd7a91a9edcac7c80/salt/utils/event.py#L823-L829

For fatal errors, the state module usually returns lists (rendering errors)/strings (uncaught exceptions), which end up in return.

[morenski]

@lkubb Even for informational messages, the state module can return lists, e.g.:

[me@salt-master]# salt-run state.event pretty=True
[...]
salt/job/[jid]/ret/[minion_id]  {
    "_stamp": "2024-01-12T00:33:39.215860",
    "arg": [
        "[arg1]"
    ],
    "cmd": "_return",
    "fun": "state.sls",
    "fun_args": [
        "[arg1]"
    ],
    "id": "[minion_id]",
    "jid": "[jid]",
    "out": "highstate",
    "pid": 12270,
    "retcode": 1,
    "return": [
        "The function \"state.sls\" is running as PID 12250 and was started at 2024, Jan 12 00:33:36.053637 with jid [jid]"
    ],
    "schedule": "[schedule1]",
    "success": true,
    "tgt": "[minion_id]",
    "tgt_type": "glob"
}
[...]

The salt master's log file shows the following at the same time (in a different time zone):

2024-01-11 16:33:39,216 [salt.utils.event :912 ][ERROR   ][2229253] Event iteration failed with exception: 'list' object has no attribute 'items'

[lkubb]

Yeah, in my mind I had classified this as a rendering error since no states ran. I wouldn't call it informational though, it's failing the requested execution for good.

Either way, the linked code should handle lists and strings in return as well, which would likely fix this issue.

[lomeroe]

looks like this is a side effect of #60782 which added the error message versus the "pass" that had been in the exception block for ages (back to at least v2016.11) in the _fire_ret_load_specific_fun function

we're using the following patch to suppress the error again:

--- /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/event.py        2023-10-16 12:22:41.000000000 -0500
+++ /tmp/event.py       2024-01-24 09:08:48.923528695 -0600
@@ -890,24 +890,25 @@
             retcode = load["retcode"]

         try:
-            for tag, data in ret.items():
-                data["retcode"] = retcode
-                tags = tag.split("_|-")
-                if data.get("result") is False:
-                    self.fire_event(
-                        data, "{}.{}".format(tags[0], tags[-1])
-                    )  # old dup event
-                    data["jid"] = load["jid"]
-                    data["id"] = load["id"]
-                    data["success"] = False
-                    data["return"] = "Error: {}.{}".format(tags[0], tags[-1])
-                    data["fun"] = fun
-                    if "user" in load:
-                        data["user"] = load["user"]
-                    self.fire_event(
-                        data,
-                        tagify([load["jid"], "sub", load["id"], "error", fun], "job"),
-                    )
+            if getattr(ret, "items", None):
+                for tag, data in ret.items():
+                    data["retcode"] = retcode
+                    tags = tag.split("_|-")
+                    if data.get("result") is False:
+                        self.fire_event(
+                            data, "{}.{}".format(tags[0], tags[-1])
+                        )  # old dup event
+                        data["jid"] = load["jid"]
+                        data["id"] = load["id"]
+                        data["success"] = False
+                        data["return"] = "Error: {}.{}".format(tags[0], tags[-1])
+                        data["fun"] = fun
+                        if "user" in load:
+                            data["user"] = load["user"]
+                        self.fire_event(
+                            data,
+                            tagify([load["jid"], "sub", load["id"], "error", fun], "job"),
+                        )
         except Exception as exc:  # pylint: disable=broad-except
             log.error(
                 "Event iteration failed with exception: %s",

likely should be handled differently, as @lkubb described, to still fire an event on the errors that are returned as lists/strings/etc, but from our perspective, removing the log "spam" and returning to the previous state before #60782 were good enough for us at this time

[dwoz]

@morenski File client timed out after 1691084572 indicates that the fileclient has timed out. You can adjust this timeout with the request_channel_timeout and request_channel_tries minion config option.