salt-minion 3003 not starting with system.daemon python error

AxisNL commented 3 years ago

Description of Issue

Hey, I'm managing a whole bunch of Ubuntu machines, and all of a sudden (I think after an automatic upgrade), salt-minion won't start anymore on my minions:

# /usr/bin/salt-minion --version
Traceback (most recent call last):
  File "/usr/bin/salt-minion", line 11, in <module>
    load_entry_point('salt==3003', 'console_scripts', 'salt-minion')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 181, in salt_minion
    salt.utils.process.notify_systemd()
  File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 184, in notify_systemd
    if systemd.daemon.booted():
AttributeError: module 'systemd.daemon' has no attribute 'booted'

All machines are running Ubuntu 20.04.2 LTS, full updated.

On other machines running all the same versions, everything works fine:

# salt-minion --version
salt-minion 3003

I removed the salt-packages, rm -rf /etc/salt, and reinstalled, same issue.

I installed salt from my locally replicated mirror.

Setup

No configs yet

Steps to Reproduce Issue

run salt-minion --version

Versions Report

even /usr/bin/salt-minion --versions-report

welcome[bot] commented 3 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

AxisNL commented 3 years ago

versions on faulty machine (same as on other machines)

# dpkg -l | grep salt
ii  salt-common                          3003+ds-1                          all          shared libraries that salt requires for all packages
iF  salt-minion                          3003+ds-1                          all          client package for salt, the distributed remote execution system

# dpkg -l | grep systemd
ii  dbus-user-session                    1.12.16-2ubuntu2.1                 amd64        simple interprocess messaging system (systemd --user integration)
ii  libnss-systemd:amd64                 245.4-4ubuntu3.6                   amd64        nss module providing dynamic user and group name resolution
ii  libpam-systemd:amd64                 245.4-4ubuntu3.6                   amd64        system and service manager - PAM module
ii  libsystemd-dev:amd64                 245.4-4ubuntu3.6                   amd64        systemd utility library - development files
ii  libsystemd0:amd64                    245.4-4ubuntu3.6                   amd64        systemd utility library
ii  networkd-dispatcher                  2.0.1-1                            all          Dispatcher service for systemd-networkd connection status changes
ii  python3-systemd                      234-3build2                        amd64        Python 3 bindings for systemd
ii  systemd                              245.4-4ubuntu3.6                   amd64        system and service manager
ii  systemd-sysv                         245.4-4ubuntu3.6                   amd64        system and service manager - SysV links
rc  systemd-timesyncd                    245.4-4ubuntu3.4                   amd64        minimalistic service to synchronize local time with NTP servers

OrangeDog commented 3 years ago

--versions-report also being broken is annoying. Have you tried salt-call --local --versions?

I have the same versions of everything else you've mentioned and it's all fine.

``` Salt Version: Salt: 3003 Dependency Versions: cffi: 1.14.5 cherrypy: Not Installed dateutil: 2.7.3 docker-py: Not Installed gitdb: 2.0.6 gitpython: 3.0.7 Jinja2: 2.10.1 libgit2: 1.1.0 M2Crypto: 0.31.0 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.6.1 pygit2: 1.5.0 Python: 3.8.5 (default, Jan 27 2021, 15:41:15) python-gnupg: 0.4.5 PyYAML: 5.3.1 PyZMQ: 18.1.1 smmap: 2.0.5 timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.2 System Versions: dist: ubuntu 20.04 focal locale: utf-8 machine: x86_64 release: 5.4.0-73-generic system: Linux version: Ubuntu 20.04 focal ```

garethgreenaway commented 3 years ago

@AxisNL Thanks for the report. To make sure we're able to fix this are you using the Salt packages provided by the Salt Project or the packages provided by Ubuntu?

AxisNL commented 3 years ago

That last command seems to give output! It is identical to the working node though!

I'm using the packages from the salt repo (https://s3.repo.saltproject.io synced to local mirror on our air-gapped networks.

# salt-call --local --versions
Salt Version:
          Salt: 3003

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.10.1
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.6.1
        pygit2: Not Installed
        Python: 3.8.5 (default, Jan 27 2021, 15:41:15)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-73-generic
        system: Linux
       version: Ubuntu 20.04 focal

# salt-minion --versions-report
Traceback (most recent call last):
  File "/usr/bin/salt-minion", line 11, in <module>
    load_entry_point('salt==3003', 'console_scripts', 'salt-minion')()
  File "/usr/lib/python3/dist-packages/salt/scripts.py", line 181, in salt_minion
    salt.utils.process.notify_systemd()
  File "/usr/lib/python3/dist-packages/salt/utils/process.py", line 184, in notify_systemd
    if systemd.daemon.booted():
AttributeError: module 'systemd.daemon' has no attribute 'booted'

AxisNL commented 3 years ago

good node:

# python3
Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import systemd.daemon
>>> dir(systemd.daemon)
['LISTEN_FDS_START', '_AF_UNSPEC', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_convert_fileobj', '_is_fifo', '_is_mq', '_is_socket', '_is_socket_inet', '_is_socket_sockaddr', '_is_socket_unix', '_listen_fds', 'booted', 'is_fifo', 'is_mq', 'is_socket', 'is_socket_inet', 'is_socket_sockaddr', 'is_socket_unix', 'listen_fds', 'notify']
>>> 

# pip freeze | grep systemd
systemd-python==234

bad node:

# python3
Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import systemd.daemon
>>> dir(systemd.daemon)
['LISTEN_FDS_START', '_AF_UNSPEC', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', '__version__', '_convert_fileobj', '_is_fifo', '_is_mq', '_is_socket', '_is_socket_inet', '_is_socket_sockaddr', '_is_socket_unix', '_listen_fds', 'booted', 'is_fifo', 'is_mq', 'is_socket', 'is_socket_inet', 'is_socket_sockaddr', 'is_socket_unix', 'listen_fds', 'notify']
>>> 

# pip freeze | grep systemd
systemd==0.16.1
systemd-python==234

After seeing this, I removed the systemd package (no idea how that got installed), salt-minion suddenly works fine! So it looks like salt was using the systemd package instead of the systemd-python package.

This machine was clean installed a few weeks back, running graylog (+mongodb +elasticsearch). Perhaps one of those packages installed the systemd package.

Well, anyway, it turned out not to be related to salt. Although perhaps an extra check could be made that it's importing from pip package systemd-python instead of systemd, don't know if that's possible.

saltstack / salt