Open asingh3-vmware opened 1 year ago
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!
Does the problem still occur with 3004.2?
You also say you get a timeout error, but the log shows a StreamClosedError
instead.
we are in process of upgrading to 3004.2, but wanted to understand in which scenario this issue can occur.
File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event self.pusher.send(msg)
Judging by the stack trace, that issue was fixed by https://github.com/saltstack/salt/pull/62952
Looks like this or similar is an issue in 3005.1. My minions randomly stop communicating and then I have to restart the processes to get them to work.
[lbird@gcp3476prdamn02 ~]$ sudo salt-call test.ping
[ERROR ] Caught exception in PubChannel connect callback StreamClosedError('Stream is closed',)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/channel/client.py", line 487, in connect_callback
self.event.fire_event({"master": self.opts["master"]}, "__master_connected")
File "/usr/lib/python3.6/site-packages/salt/utils/event.py", line 833, in fire_event
self.pusher.send(msg)
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
raise exc_info[1].with_traceback(exc_info[2])
File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
return future_cell[0].result()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "<string>", line 4, in raise_exc_info
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
yielded = self.gen.send(value)
File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 456, in send
yield self.stream.write(pack)
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/iostream.py", line 388, in write
self._check_closed()
File "/usr/lib/python3.6/site-packages/salt/ext/tornado/iostream.py", line 926, in _check_closed
raise StreamClosedError(real_error=self.error)
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
[lbird@gcp3476prdamn02 ~]$ sudo salt --versions-report
Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: 1.14.5
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: 4.4.4
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: 3.10.1
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.90.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
seeing this on salt master salt.ext.tornado.iostream.StreamClosedError: Stream is closed
salt --versions-report
Salt Version:
Salt: 3006.3
Python Version:
Python: 3.10.4 (main, Apr 20 2022, 01:21:48) [GCC 10.3.1 20210424]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.11
gitpython: 3.1.41
Jinja2: 3.1.2
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: Not Installed
smmap: 5.0.1
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: alpine 3.14.6
locale: utf-8
machine: x86_64
release: 5.15.0-1055-aws
system: Linux
version: Alpine Linux 3.14.6
Description Salt Minion not able to Connect with Salt Master. Getting Timeout error.
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
Steps to Reproduce the behavior 2022-11-05 23:04:39,040 [tornado.application :356 ][ERROR ][52] Future <salt.ext.tornado.concurrent.Future object at 0x7f10670c7790> exception was never retrieved: Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper yielded = next(result) File "/usr/lib/python3.7/site-packages/salt/transport/zeromq.py", line 780, in handle_message stream.send(salt.payload.dumps(self._auth(payload["load"], sign_messages))) File "/usr/lib/python3.7/site-packages/salt/transport/mixins/auth.py", line 439, in _auth eload, salt.utils.event.tagify(prefix="auth") File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event self.pusher.send(msg) File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 125, in wrap raise exc_info[1].with_traceback(exc_info[2]) File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 131, in _target result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, *kwargs)) File "/usr/lib/python3.7/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync return future_cell[0].result() File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1064, in run
yielded = self.gen.throw( exc_info)
File "/usr/lib/python3.7/site-packages/salt/transport/ipc.py", line 453, in send
yield self.stream.write(pack)
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1056, in run
value = future.result()
File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
raise_exc_info(self._exc_info)
File "", line 4, in raise_exc_info
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
Expected behavior
Minion should able to connect with master, and existing connection b/w master-minion should continue.
Screenshots
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml root [ / ]# salt -V Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.11.5 cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 0.5.6 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.18 pycrypto: Not Installed pycryptodome: 3.9.9 pygit2: Not Installed Python: 3.7.5 (default, Oct 4 2022, 20:43:27) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.1.2 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: photon 3.0 Photon locale: utf-8 machine: x86_64 release: 4.19.256-2.ph3 system: Linux version: VMware Photon OS 3.0 Photon ```Additional context Minion is also using same version of master. neither new minion is able to establish connection not master is able to retain existing connection.