[BUG] Unable to forward pub data (running salt-syndic on a container)

[neilspage2]

[WARNING ] Unable to forward pub data: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass the congested event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

This is what I get from the podman container.

However, from the container I am able to hit the master AND the syndic:

# syndic node: master daemon
[root@f175df9f60a8 data]# nc -zv <omitted> 4505
Connection to  <omitted> ( <omitted>) 4505 port [tcp/*] succeeded!
[root@f175df9f60a8 data]#

# master of masters
**[root@f175df9f60a8 data]# nc -zv <omitted> 4505
Connection to <omitted> (<omitted>) 4505 port [tcp/*] succeeded!**

Salt-master on the syndic node shows it's able to process jobs:

[ERROR   ] ReqServer workers ipc:///var/run/salt/master/workers.ipc
[INFO    ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[INFO    ] Worker binding to socket ipc:///var/run/salt/master/workers.ipc
[INFO    ] Authentication request from minion-host.somefqdn.com
[INFO    ] Authentication accepted from minion-host.somefqdn.com
[INFO    ] Got return from minion-host.somefqdn.com for job 20221219103820346896
[INFO    ] Authentication request from minion-host.somefqdn.com
[INFO    ] Authentication accepted from minion-host.somefqdn.com
[INFO    ] Got return from minion-host.somefqdn.com for job 20221219103835854268

The syndic component of the syndic node does not return back to the minions however, here are some errors from it:

[ERROR   ] Future <salt.ext.tornado.concurrent.Future object at 0x7fffded03f40> exception was never retrieved: Traceback (most recent call last):
  File "salt/ext/tornado/gen.py", line 1064, in run                                                                                                                             yielded = self.gen.throw(*exc_info)                                                                                                                                       File "salt/channel/client.py", line 426, in wrap_callback
    decoded = yield self._decode_payload(payload)                                                                                                                             File "salt/ext/tornado/gen.py", line 1056, in run                                                                                                                             value = future.result()
  File "salt/ext/tornado/concurrent.py", line 249, in result                                                                                                                    raise_exc_info(self._exc_info)                                                                                                                                            File "<string>", line 4, in raise_exc_info
  File "salt/ext/tornado/gen.py", line 309, in wrapper                                                                                                                          yielded = next(result)                                                                                                                                                    File "salt/channel/client.py", line 554, in _decode_payload
    if payload["enc"] == "aes":                                                                                                                                             TypeError: 'NoneType' object is not su

And

[WARNING ] Unable to forward pub data: Salt request timed out. The master is not responding. You may need to run your command with `--async` in order to bypass t[702/94993]
d event bus. With `--async`, the CLI tool will print the job id (jid) and exit immediately without listening for responses. You can then use `salt-run jobs.lookup_jid` to look up the results of the job in the job cache later.

[welcome[bot]]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

Possible duplicate: #62577