Dockerng failing to import docker on fresh salt-cloud instances

[tyhunt99]

Description of Issue/Question

I am launching a new EC2 instance through salt-cloud (via the api). I have version pinned salt to 2015.8.7 for both master and minion. Whenever I try to do anything with the Dockerng module it fails with an import error.

The strange part is as soon as restart the salt-minion it begins to work just fine.

Steps to Reproduce Issue

Launch an EC2 instance thorugh salt cloud specifying script args: -P 2015.8.7 the cloud providers definition.

Then ensure docker and docker-py is running on the new instance. I do this through the docker formula provided by salt.

Then simply try to run any Dockerng commands e.g.:

salt 'test2' dockerng.pull alpine
test2:
    'dockerng.pull' is not available

and you will get the following error in the salt-minion log:

Failed to import module dockerng, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1135, in _load_module
    ), fn_, fpath, desc)
  File "/usr/lib/python2.7/dist-packages/salt/modules/dockerng.py", line 272, in <module>
    import docker
  File "/usr/local/lib/python2.7/dist-packages/docker/__init__.py", line 20, in <module>
    from .client import Client, AutoVersionClient, from_env # flake8: noqa
  File "/usr/local/lib/python2.7/dist-packages/docker/client.py", line 25, in <module>
    from . import api
  File "/usr/local/lib/python2.7/dist-packages/docker/api/__init__.py", line 2, in <module>
    from .build import BuildApiMixin
  File "/usr/local/lib/python2.7/dist-packages/docker/api/build.py", line 9, in <module>
    from .. import utils
  File "/usr/local/lib/python2.7/dist-packages/docker/utils/__init__.py", line 1, in <module>
    from .utils import (
  File "/usr/local/lib/python2.7/dist-packages/docker/utils/utils.py", line 33, in <module>
    from .. import tls
  File "/usr/local/lib/python2.7/dist-packages/docker/tls.py", line 5, in <module>
    from .ssladapter import ssladapter
  File "/usr/local/lib/python2.7/dist-packages/docker/ssladapter/__init__.py", line 1, in <module>
    from .ssladapter import SSLAdapter # flake8: noqa
  File "/usr/local/lib/python2.7/dist-packages/docker/ssladapter/ssladapter.py", line 22, in <module>
    urllib3.connection.match_hostname = match_hostname
AttributeError: 'module' object has no attribute 'connection'

Then restarting the salt minion will make this issue go away:

 salt 'test2' dockerng.pull alpine
test2:
    ----------
    Layers:
        ----------
        Pulled:
            - 3e467a6273a3
    Status:
        Downloaded newer image for alpine:latest
    Time_Elapsed:
        0.799014091492

Versions Report

master:

Salt Version:
           Salt: 2015.8.7

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: 0.9.1
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
           cffi: 1.5.0
       cherrypy: 3.2.2
       dateutil: 2.4.2
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
        libgit2: 0.23.4
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: 1.2.3
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.23.1
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 14.04 trusty
        machine: x86_64
        release: 3.13.0-71-generic
         system: Ubuntu 14.04 trusty

minion:

    Salt Version:
               Salt: 2015.8.7

    Dependency Versions:
             Jinja2: 2.7.2
           M2Crypto: 0.21.1
               Mako: Not Installed
             PyYAML: 3.10
              PyZMQ: 14.0.1
             Python: 2.7.6 (default, Jun 22 2015, 17:58:13)
               RAET: Not Installed
            Tornado: 4.3
                ZMQ: 4.0.4
               cffi: Not Installed
           cherrypy: Not Installed
           dateutil: Not Installed
              gitdb: Not Installed
          gitpython: Not Installed
              ioflo: Not Installed
            libgit2: Not Installed
            libnacl: Not Installed
       msgpack-pure: Not Installed
     msgpack-python: 0.3.0
       mysql-python: Not Installed
          pycparser: Not Installed
           pycrypto: 2.6.1
             pygit2: Not Installed
       python-gnupg: Not Installed
              smmap: Not Installed
            timelib: Not Installed

    System Versions:
               dist: Ubuntu 14.04 trusty
            machine: x86_64
            release: 3.13.0-74-generic
             system: Ubuntu 14.04 trusty

[tyhunt99]

I have also seen the same issues when using salt 2015.8.10.

[ticosax]

Hi, This is docker-py issue I believe docker/docker-py#1054

[Ch3LL]

@tyhunt99 as @ticosax mentioned this is possibly a docker-py issue. Can you open a python interpreter and run from docker import Client from that machine?

Also can you try updating your version of urlib3 as the issue states? Also possibly trying to upgrade or downgrade your docker-py will help to determine if this upstream or not

[abednarik]

HI

Are you also doing a reload_modules: True after installing docker-py? Also make sure to have a recent pip version like 8.1.1.

[mitar]

My issue was that a new version of docker-py got installed (2.0) because a newer version of docker-compose (1.10.0) got installed which forced this new version of docker-py. And docker-py does not have a Client.

[alem0lars-yr]

Any updates ?

[Ch3LL]

@alem0lars-yoroi are you running into the same thing? what update are you looking for specifically?

We need to hear from @tyhunt99 to see if he was able to resovle this issue with the posted solutions above

[alem0lars-yr]

@Ch3LL The salt minion doesn't get the docker-py, thus issues an error when a state requires it. I've already tried to use reload_modules: True.

This is the relevant part of my sls:

{# == DOCKER REPOSITORY ===================================================== #}

docker-package-repository:
  pkgrepo.managed:
    - humanname: {{ grains['os'] }} {{ grains['oscodename'] | capitalize }} docker package repository
    - name: 'deb https://apt.dockerproject.org/repo {{ grains['os'] | lower }}-{{ grains['oscodename'] }} main'
    - keyid: 58118E89F3A912897C070ADBF76221572C52609D
    - keyserver: hkp://p80.pool.sks-keyservers.net:80
    - file: /etc/apt/sources.list.d/docker.list
    - refresh_db: True

{# == DOCKER SERVICE ======================================================== #}

docker-package-dependencies:
  pkg.installed:
    - pkgs:
      - apt-transport-https
      - ca-certificates
      - curl
      - software-properties-common
      - python-software-properties

docker-package:
  pkg.installed:
    - pkgs:
      - docker-engine
    - require:
      - pkg: docker-package-dependencies
      - pkgrepo: docker-package-repository

docker-config:
  file.managed:
    - name: /etc/default/docker
    - source: salt://honey/docker/files/config
    - template: jinja
    - mode: 644
    - user: root

docker-service:
  service.running:
    - name: docker
    - enable: True
    - sig: /usr/bin/docker
    - watch:
      - file: /etc/default/docker
      - pkg: docker-package

{# == DOCKER-PY ============================================================= #}

python-pip:
  pkg:
    - installed

docker-py-dependencies:
  pip.installed:
    - name: pip
    - upgrade: True
    - require:
      - pkg: python-pip

docker-py:
  pip.installed:
    - name: docker-py
    - upgrade: True
    - reload_modules: True # Somehow it is not working!
    - require:
      - pip: docker-py-dependencies

{# ========================================================================== #}

Then when I try to use dockerng (that requires docker-py) it gives me the following error:

Could not LazyLoad dockerng.running: 'dockerng' __virtual__ returned False: 'dockerng' __virtual__ returned False: Docker module could not get imported; You need to "pip install docker-py"

[tyhunt99]

Sorry this issue slipped through the cracks. But as of 2016.11.3 I am still seeing this issue. I am using the docker formula to set up docker and then trying to start a container.

The relevant errors I get:

'pip_|-docker-py_|-docker-py_|-installed':{ 
      'duration':924.653,
      '__id__':'docker-py',
      'comment':'Python package docker-py was already installed\nAll packages were successfully installed',
      '__run_num__':9,
      'name':'docker-py',
      'changes':{

      },
      'start_time':'18:19:46.167870',
      'result':True
    },
'dockerng_|-autodns_image_|-364535168108.dkr.ecr.us-east-1.amazonaws.com/autodns:latest_|-image_present':{ 
      '__run_num__':12,
      'name':'364535168108.dkr.ecr.us-east-1.amazonaws.com/autodns:latest',
      'changes':{ 

      },
      'result':False,
      'comment':'State \'dockerng.image_present\' was not found in SLS \'docker_host.autodns\'\nReason: \'dockerng\' __virtual__ returned False: \'dockerng\' __virtual__ returned False: Docker module could not get imported; You need to "pip install docker-py"\n'
    }

I have a current work around. If I restart the salt-minion service it picks up docker-py is installed.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[tyhunt99]

As of salt 2018.3.2 I am no longer seeing this issue so I think it was inadvertently fixed somewhere along the line.