saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.11k stars 5.47k forks source link

Salt Connectivity issue in 3004.1 #63074

Open asingh3-vmware opened 1 year ago

asingh3-vmware commented 1 year ago

Description Salt Minion not able to Connect with Salt Master. Getting Timeout error.

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

Steps to Reproduce the behavior 2022-11-05 23:04:39,040 [tornado.application :356 ][ERROR ][52] Future <salt.ext.tornado.concurrent.Future object at 0x7f10670c7790> exception was never retrieved: Traceback (most recent call last): File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 309, in wrapper yielded = next(result) File "/usr/lib/python3.7/site-packages/salt/transport/zeromq.py", line 780, in handle_message stream.send(salt.payload.dumps(self._auth(payload["load"], sign_messages))) File "/usr/lib/python3.7/site-packages/salt/transport/mixins/auth.py", line 439, in _auth eload, salt.utils.event.tagify(prefix="auth") File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event self.pusher.send(msg) File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 125, in wrap raise exc_info[1].with_traceback(exc_info[2]) File "/usr/lib/python3.7/site-packages/salt/utils/asynchronous.py", line 131, in _target result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, *kwargs)) File "/usr/lib/python3.7/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync return future_cell[0].result() File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1064, in run yielded = self.gen.throw(exc_info) File "/usr/lib/python3.7/site-packages/salt/transport/ipc.py", line 453, in send yield self.stream.write(pack) File "/usr/lib/python3.7/site-packages/salt/ext/tornado/gen.py", line 1056, in run value = future.result() File "/usr/lib/python3.7/site-packages/salt/ext/tornado/concurrent.py", line 249, in result raise_exc_info(self._exc_info) File "", line 4, in raise_exc_info salt.ext.tornado.iostream.StreamClosedError: Stream is closed Expected behavior Minion should able to connect with master, and existing connection b/w master-minion should continue.

Screenshots

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml root [ / ]# salt -V Salt Version: Salt: 3004.1 Dependency Versions: cffi: 1.11.5 cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 0.5.6 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.18 pycrypto: Not Installed pycryptodome: 3.9.9 pygit2: Not Installed Python: 3.7.5 (default, Oct 4 2022, 20:43:27) python-gnupg: Not Installed PyYAML: 3.13 PyZMQ: 17.1.2 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: photon 3.0 Photon locale: utf-8 machine: x86_64 release: 4.19.256-2.ph3 system: Linux version: VMware Photon OS 3.0 Photon ```

Additional context Minion is also using same version of master. neither new minion is able to establish connection not master is able to retain existing connection.

welcome[bot] commented 1 year ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

OrangeDog commented 1 year ago

Does the problem still occur with 3004.2?

You also say you get a timeout error, but the log shows a StreamClosedError instead.

asingh3-vmware commented 1 year ago

we are in process of upgrading to 3004.2, but wanted to understand in which scenario this issue can occur.

frebib commented 1 year ago

File "/usr/lib/python3.7/site-packages/salt/utils/event.py", line 829, in fire_event self.pusher.send(msg)

Judging by the stack trace, that issue was fixed by https://github.com/saltstack/salt/pull/62952

TheBirdsNest commented 1 year ago

Looks like this or similar is an issue in 3005.1. My minions randomly stop communicating and then I have to restart the processes to get them to work.

[lbird@gcp3476prdamn02 ~]$ sudo salt-call test.ping
[ERROR   ] Caught exception in PubChannel connect callback StreamClosedError('Stream is closed',)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/channel/client.py", line 487, in connect_callback
    self.event.fire_event({"master": self.opts["master"]}, "__master_connected")
  File "/usr/lib/python3.6/site-packages/salt/utils/event.py", line 833, in fire_event
    self.pusher.send(msg)
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 125, in wrap
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/lib/python3.6/site-packages/salt/utils/asynchronous.py", line 131, in _target
    result = io_loop.run_sync(lambda: getattr(self.obj, key)(*args, **kwargs))
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/ioloop.py", line 459, in run_sync
    return future_cell[0].result()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/concurrent.py", line 249, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/gen.py", line 1070, in run
    yielded = self.gen.send(value)
  File "/usr/lib/python3.6/site-packages/salt/transport/ipc.py", line 456, in send
    yield self.stream.write(pack)
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/iostream.py", line 388, in write
    self._check_closed()
  File "/usr/lib/python3.6/site-packages/salt/ext/tornado/iostream.py", line 926, in _check_closed
    raise StreamClosedError(real_error=self.error)
salt.ext.tornado.iostream.StreamClosedError: Stream is closed
TheBirdsNest commented 1 year ago
[lbird@gcp3476prdamn02 ~]$ sudo salt --versions-report
Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.14.5
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: 4.4.4
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: 3.10.1
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 18.0.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: centos 7 Core
        locale: UTF-8
       machine: x86_64
       release: 3.10.0-1160.90.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7 Core
ctmackay commented 6 months ago

seeing this on salt master salt.ext.tornado.iostream.StreamClosedError: Stream is closed

salt --versions-report
Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.4 (main, Apr 20 2022, 01:21:48) [GCC 10.3.1 20210424]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.11
     gitpython: 3.1.41
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: Not Installed
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: alpine 3.14.6
        locale: utf-8
       machine: x86_64
       release: 5.15.0-1055-aws
        system: Linux
       version: Alpine Linux 3.14.6