swarm module init exception during lazyload

balrawat commented 5 years ago

Description of Issue

I'm getting exception in the debug logs during lazyloading of swarm module. swarm or docker is not used in my setup/configs/states. This is not a blocker for me but it might be hitting some edge case and help in cleaning logs as well.

[DEBUG   ] Error loading module.swarm: __init__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1624, in _load_module
    module_init(self.opts)
  File "/usr/lib/python3.6/site-packages/salt/modules/swarm.py", line 53, in __init__
    __context__['client'] = docker.from_env()
AttributeError: module 'docker' has no attribute 'from_env'

Setup

I've tested it on freshly provision google compute VM ( n1-standard-1 (1 vCPU, 3.75 GB memory) ) OS: CentOS Linux release 7.7.1908 (Core) Kernel: 3.10.0-1062.1.2.el7.x86_64

Minion config:

[root@salt1 global]# cat /etc/salt/minion
file_roots:
  base:
    - /srv/salt/global

pillar_roots:
  base:
    - /srv/pillar

States:

[root@salt1 global]# cat top.sls
base:
  '*':
    - mystate
[root@salt1 global]# cat mystate.sls
sshd_config_restart:
  service.running:
    - name: sshd
    - enable: True
    - onlyif:
      - sshd -t
    - check_cmd:
      - sshd -t

{% for package in ['git','bzip2','gcc-c++','make'] %}
pkg-{{ package }}:
  pkg.installed:
    - name: {{ package }}
{% endfor %}

Steps to Reproduce Issue

Using the above configs/states run: salt-call state.highstate --local -l debug

Versions Report

[root@salt1 global]# salt-call --versions-report
Salt Version:
           Salt: 2019.2.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.33.0
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.8 (default, Aug  7 2019, 17:28:10)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.7.1908 Core
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.10.0-1062.1.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.7.1908 Core

balrawat commented 5 years ago

It occurs during a module refresh after any change happens.

log snippet:

[INFO    ] Made the following changes:
'bzip2' changed from 'absent' to '1.0.6-13.el7'

[DEBUG   ] Refreshing modules...
[INFO    ] Loading fresh modules for state activity
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[INFO    ] Completed state [bzip2] at time 18:09:06.789075 (duration_in_ms=13877.258)
[DEBUG   ] LazyLoaded config.option
[DEBUG   ] LazyLoaded pkg.install
[DEBUG   ] LazyLoaded pkg.installed
[INFO    ] Running state [gcc-c++] at time 18:09:06.810082
[INFO    ] Executing state pkg.installed for [gcc-c++]
[DEBUG   ] key: ifttt.secret_key, ret: _|-
[DEBUG   ] key: ifttt:secret_key, ret: _|-
[DEBUG   ] key: pushbullet.api_key, ret: _|-
[DEBUG   ] key: pushbullet:api_key, ret: _|-
[DEBUG   ] LazyLoaded systemd.booted
**[DEBUG   ] Error loading module.swarm: __init__ failed
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1624, in _load_module
    module_init(self.opts)
  File "/usr/lib/python3.6/site-packages/salt/modules/swarm.py", line 53, in __init__
    __context__['client'] = docker.from_env()
AttributeError: module 'docker' has no attribute 'from_env'**
[DEBUG   ] key: victorops.api_key, ret: _|-
[DEBUG   ] key: victorops:api_key, ret: _|-

[DEBUG   ] DSC: Only available on Windows systems
[DEBUG   ] Module PSGet: Only available on Windows systems
[DEBUG   ] LazyLoaded platform.is_windows
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [gcc-c++] at time 18:09:12.790455 (duration_in_ms=5980.372)
[INFO    ] Running state [make] at time 18:09:12.790690
[INFO    ] Executing state pkg.installed for [make]
[DEBUG   ] Could not LazyLoad pkg.resolve_capabilities: 'pkg.resolve_capabilities' is not available.
[INFO    ] All specified packages are already installed
[INFO    ] Completed state [make] at time 18:09:12.814836 (duration_in_ms=24.144)
[DEBUG   ] File /var/cache/salt/minion/accumulator/140376956609144 does not exist, no need to cleanup
[DEBUG   ] LazyLoaded state.check_result
[DEBUG   ] LazyLoaded highstate.output
[DEBUG   ] LazyLoaded nested.output

whytewolf commented 5 years ago

So I dug around a while. I was able to replicate using the method described. I put in a little debugging to see what I can find.

It looks like it is importing docker from

/usr/lib/python3.6/site-packages/salt/utils/docker/__init__.py

Not sure how PYTHONPATH is getting twisted enough to load that module as if it was the docker SDK.

Ch3LL commented 5 years ago

looks like im also able to replicate this.

like you said its the docker/__init__.py file that is being imported as docker, but i do not see /root/salt/salt/utils in the path, but it was added to the path previously in the loader when we are loading up the __utils__.

>>> sys.path                                                                                                                                                                         
['/usr/local/bin', '/salt', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload',
'/usr/local/lib64/python3.6/site-packages', '/usr/local/lib/python3.6/site-packages', '/root/salt',
'/usr/lib64/python3.6/site-packages', '/usr/lib/python3.6/site-packages', '/root/salt/salt/modules']
>>> docker.__file__
'/root/salt/salt/utils/docker/__init__.py'

still looking into how to solve this.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

whytewolf commented 4 years ago

This is not stale

stale[bot] commented 4 years ago

Thank you for updating this issue. It is no longer marked as stale.

fzipi commented 4 years ago

Still happening.

ScriptAutomate commented 4 years ago

Same Issue

I'm getting the same debug chatter, much of which is entirely unrelated to what's being installed/used by pkg.install on ubuntu bionic.

References appear for the following, even though they have nothing to do with the installation of target package(s):

ifttt
pushbullet
victorops
docker swarm
DSC / PSGet (Windows OS related info)

Which are all reflected in the debug output of this issue, too? I'm not sure which are meant to be expected within default salt-call runs associated with pkg.*

[DEBUG ] key: ifttt.secret_key, ret: _|-
[DEBUG ] key: ifttt:secret_key, ret: _|-
[DEBUG ] key: pushbullet.api_key, ret: _|-
[DEBUG ] key: pushbullet:api_key, ret: _|-
[DEBUG ] LazyLoaded systemd.booted
[DEBUG ] Error loading module.swarm: init failed
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/loader.py", line 1624, in _load_module
module_init(self.opts)
File "/usr/lib/python3.6/site-packages/salt/modules/swarm.py", line 53, in init
context['client'] = docker.from_env()
AttributeError: module 'docker' has no attribute 'from_env'
[DEBUG ] key: victorops.api_key, ret: _|-
[DEBUG ] key: victorops:api_key, ret: _|-

[DEBUG ] DSC: Only available on Windows systems
[DEBUG ] Module PSGet: Only available on Windows systems

Also boto3 / AWS attempted calls?

Though, I also get boto3 (AWS) related errors in debug output, too:

[DEBUG   ] Could not LazyLoad boto3.assign_funcs: 'boto3.assign_funcs' is not available.
[DEBUG   ] Error loading module.boto3_elasticsearch: __init__ failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1664, in _load_module
    module_init(self.opts)
  File "/usr/lib/python2.7/dist-packages/salt/modules/boto3_elasticsearch.py", line 92, in __init__
    __utils__['boto3.assign_funcs'](__name__, 'es')
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1248, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 108, in __getitem__
    raise KeyError(key)
KeyError: u'boto3.assign_funcs'

My Setup

vagrant@ubuntu-bionic:~$ salt-call --versions-report
Salt Version:
           Salt: 3000.2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.17 (default, Apr 15 2020, 17:20:14)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-99-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Steps to Reproduce

Using Vagrant and Virtualbox:

vagrant box add ubuntu/bionic64 --provider virtualbox
Masterless salt setup via Vagrant Salt directions

NOTE: This uses a salt bootstrap which seems to default to using a Python 2.7.17 version of Salt 3000.2 at the moment
vagrant up
vagrant ssh
sudo salt-call --local state.apply -l debug

Salt State

vagrant@ubuntu-bionic:/srv/salt$ cat top.sls
base:
  '*':
    - git
vagrant@ubuntu-bionic:/srv/salt$ cat git.sls
git:
  pkg:
    - installed

These additional debug problems do only occur in:

sudo salt-call --local state.apply -l debug

I've only tested locally against a masterless salt minion.

When attempting to install via sudo salt-call --local pkg.install git, none of these problems happen. It seems to be tied specifically to salt states.

jwolfe-ns commented 4 years ago

This is still present in 3001. It makes parsing the debug logs a bit more work.

doesitblend commented 4 years ago

ZD-5767

sagetherage commented 4 years ago

Since this is larger than the initial title suggests and as we use docker more and more I am moving it up in severity. Also, in an attempt to get it looked at by anyone willing to open a PR against this.

saltstack / salt