search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.18k stars 5.48k forks source link

[BUG] salt-minion[xx]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_c305107a7a_pub.ipc ssl=None #66179

Open tjyang opened 8 months ago

tjyang commented 8 months ago

Description

After upgrade to 3007 and cleanup .pyc per https://github.com/saltstack/salt/issues/66177 still seeing this error message 

salt-minion[xx]: [ERROR ] Publish server binding pub to /var/run/salt/minion/minion_event_c305107a7a_pub.ipc ssl=None

Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

Steps to Reproduce the behavior

get and run bootstrap-salt.sh on existing system as root.

Expected behavior

bootstrap.sh will upgrade existing minion to latest version (3007 in this case) without error when starting up 3007 minion.

Screenshots [root@rocky8t03 ~]# systemctl status -l salt-minion ● salt-minion.service - The Salt Minion Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2024-03-06 19:44:56 CST; 1min 28s ago Docs: man:salt-minion(1) file:///usr/share/doc/salt/html/contents.html https://docs.saltproject.io/en/latest/contents.html Main PID: 969955 (python3.10) Tasks: 11 (limit: 48605) Memory: 69.6M CGroup: /system.slice/salt-minion.service ├─969955 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion └─969963 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager MinionProcessManager

Mar 06 19:44:59 rocky8t03 salt-minion[969963]: c '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault' Mar 06 19:44:59 rocky8t03 salt-minion[969963]: [ERROR ] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault' Mar 06 19:44:59 rocky8t03 salt-minion[969963]: [ERROR ] Module/package collision: '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/pycache/vault.cpython-310.pyc' and '/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/vault'

**Versions Report**
salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) ```yaml Salt Version: Salt: 3007.0 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: unknown dateutil: 2.8.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.15.1 smmap: Not Installed timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: rocky 8.9 Green Obsidian locale: utf-8 machine: x86_64 release: 4.18.0-513.18.1.el8_9.x86_64 system: Linux version: Rocky Linux 8.9 Green Obsidian [tjyang@ilclsalt01 salt]$ salt --versions-report Salt Version: Salt: 3007.0 Python Version: Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0] Dependency Versions: cffi: 1.16.0 cherrypy: unknown dateutil: 2.8.2 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.3 libgit2: Not Installed looseversion: 1.3.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.7 msgpack-pure: Not Installed mysql-python: Not Installed packaging: 23.1 pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.19.1 pygit2: Not Installed python-gnupg: 0.5.2 PyYAML: 6.0.1 PyZMQ: 25.1.2 relenv: 0.15.1 smmap: Not Installed timelib: 0.3.0 Tornado: 6.3.3 ZMQ: 4.3.4 Salt Package Information: Package Type: onedir System Versions: dist: rocky 8.9 Green Obsidian locale: utf-8 machine: x86_64 release: 4.18.0-513.18.1.el8_9.x86_64 system: Linux version: Rocky Linux 8.9 Green Obsidian ```
**Additional context** current workaround is to remove /opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/__pycache__ directory and restart salt-minion. but still having following errors ``` CGroup: /system.slice/salt-minion.service ├─3442349 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion └─3442363 /opt/saltstack/salt/bin/python3.10 /usr/bin/salt-minion MultiMinionProcessManager MinionProcessManager MinionProce> Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", lin> Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: await callback(decoded) Mar 07 05:20:29 va32lsalt01 salt-minion[3442363]: TypeError: object NoneType can't be used in 'await' expression Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: [ERROR ] Exception while running callback Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: Traceback (most recent call last): Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/transport/zeromq.py", l> Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: await callback(msg) Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/channel/client.py", lin> Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: await callback(decoded) Mar 07 05:20:34 va32lsalt01 salt-minion[3442363]: TypeError: object NoneType can't be used in 'await' expression ```
tjyang commented 8 months ago

This is similar to https://github.com/saltstack/salt/issues/66177

AGirin commented 8 months ago

I get similar error but on the master:

salt-master[2925533]: [ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None
nicholasmhughes commented 8 months ago

These look like they should be debug statements...

https://github.com/saltstack/salt/blob/562e739408ce4aef14400dc6aea91c12a3f5c425/salt/transport/tcp.py#L1404-L1420

@dwoz , can you confirm?

ghost commented 7 months ago

Same issue

vhumphrey commented 7 months ago

Master Description: After upgrade to 3007 and cleanup .pyc per https://github.com/saltstack/salt/issues/66177 seeing this error message:

systemctl status -l salt-master

salt-master.service - The Salt Master Server
     Loaded: loaded (/lib/systemd/system/salt-master.service; enabled; preset: enabled)
     Active: active (running) since Wed 2024-04-03 17:53:44 EDT; 8min ago
       Docs: man:salt-master(1)
             file:///usr/share/doc/salt/html/contents.html
             https://docs.saltproject.io/en/latest/contents.html
   Main PID: 1916 (/opt/saltstack/)
      Tasks: 22 (limit: 4652)
     Memory: 167.8M
        CPU: 13.569s
     CGroup: /system.slice/salt-master.service
             ├─1916 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master MainProcess"
             ├─2110 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master PubServerChannel._publish_daemon"
             ├─2111 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventPublisher"
             ├─2112 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master EventMonitor"
             ├─2115 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master Maintenance"
             ├─2116 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer ReqServer_ProcessManager"
             ├─2117 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master FileServerUpdate"
             ├─2118 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorkerQueue"
             ├─2119 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-0"
             ├─2121 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-1"
             ├─2128 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-2"
             ├─2129 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-3"
             └─2130 "/opt/saltstack/salt/bin/python3.10 /usr/bin/salt-master ReqServer MWorker-4"

Apr 03 17:52:38 vm-202311071019-debian12 systemd[1]: Starting salt-master.service - The Salt Master Server...
Apr 03 17:53:44 vm-202311071019-debian12 systemd[1]: Started salt-master.service - The Salt Master Server.
Apr 03 18:02:11 vm-202311071019-debian12 salt-master[2111]: [ERROR   ] Publish server binding pub to /var/run/salt/master/master_event_pub.ipc ssl=None

root@vm-202311071019-debian12:~# salt --versions-report

Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm
rolszewo commented 7 months ago

i've installed 3007.0 server and it gives me this error too, like here and like in #66282: [ERROR ] Publish server binding pub to /run/salt/master/master_event_pub.ipc ssl=None

i see permissions in /run/salt/master/

3005.1:

srw-rw-rw- 1 salt salt   0 Apr  4 16:16 master_event_pub.ipc
srw------- 1 salt salt   0 Apr  4 16:16 master_event_pull.ipc
srw------- 1 salt salt   0 Apr  4 16:16 publish_pull.ipc
srwxr-xr-x 1 salt salt   0 Apr  4 16:16 workers.ipc

and with master 3007.0 i see

srw------- 1 salt salt   0 Apr  4 16:26 master_event_pub.ipc
srw------- 1 salt salt   0 Apr  4 16:26 master_event_pull.ipc
srw------- 1 salt salt   0 Apr  4 16:26 publish_pull.ipc
srwx------ 1 salt salt   0 Apr  4 16:26 workers.ipc

for now i have to roll back since it seems a 3007.0 issue

kossde commented 5 months ago

I was able to 'resolve' this by granting the salt user full access to all files under /var/run/salt/master. It may be overkill, but after running 'chmod 777 /var/run/salt/master/*' and restarting salt-master, it began working as designed.

nf-brentsaner commented 5 months ago

@kossde You just gave every user full access to everything under /var/run/salt/master.

What you should have done is chown -R salt:salt /var/run/salt/master.

I recommend rebooting your master so /var/run/salt is recreated, then performing the above instead.