search

vmware-archive / salt-pack

Salt Package Builder
Apache License 2.0
55 stars 23 forks source link

Ensure python-tornado builds from SaltStack have a dependency on concurrent-futures #593

Closed dmurphy18 closed 5 years ago

dmurphy18 commented 5 years ago

Check Ubuntu 16.04 esp.

arminbuerkle commented 5 years ago

We ran into this issue today as well, when we updated our salt-minions to 2017.7.8+ds-1 on debian stretch.

Highstates resulted in an exception:

    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/dist-packages/salt/minion.py", line 1477, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/dist-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 860, in highstate
        conflict = _check_queue(queue, kwargs)
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 250, in _check_queue
        conflict = running(concurrent=kwargs.get('concurrent', False))
      File "/usr/lib/python2.7/dist-packages/salt/modules/state.py", line 209, in running
        active = __salt__['saltutil.is_running']('state.*')
      File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1121, in __getitem__
        func = super(LazyLoader, self).__getitem__(item)
      File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 101, in __getitem__
        raise KeyError(key)
    KeyError: 'saltutil.is_running'

Manually installing apt-get install -y python-concurrent.futures fixed the issue.

sblaisot commented 5 years ago

Also seen on debian 9

dmurphy18 commented 5 years ago

@sblaisot To offer a fuller explanation, SaltStack builds python-tornado for Python 2.7 which utilizes concurrent.futures (python-futures in salt-pack) for the following OS's: Raspbian 8 Debian 8 Ubuntu 16.04 Ubuntu 14.04

for other OS's the version of python-tornado (and its dependencies/recommends on concurrent.futures) is utilized. Also if one of the above OS's provides a version of python-tornado later then v4.2.1, then it is usual to utilize the later version provided by the OS.

Python 3 support for python-tornado is usually provided by the OS since SaltStack has a minimum requirement of python-tornado v4.2.1, and Python 3 implementations are typically using a later version.

Hope this clears the issue for Debian/Ubuntu OS's.

Python-tornado v4.2.1 is also built for RHEL 7 and 6, and Amazon Linux 1. There is a not a firm dependency for python-futures there, and unit test checks shall be conducted to see if it is required. Noting that EPEL is recently updated with a later version of python-tornado than that supplied by SaltStack.

dmurphy18 commented 5 years ago

Updating Debian 9 and Ubuntu 18.04 packages to require concurrent-futures to alleviate this issue. Other platforms should be unaffected.

lomeroe commented 5 years ago

@dmurphy18 I had CentOS boxes affected on fresh installs of 2018.3.3 and 2017.7.8 A manual install of python2-futures was required to get them running.

These were minimal CentOS builds with salt installed by the salt bootstrap script.

salt.utils.thin imports concurrent, which is imported by saltutil, so the python-futrues (or whatever package provides "concurrent" for a PY2 based distribution) is required to run the saltutil module

50220 has my comments on this with debug logs/etc

dmurphy18 commented 5 years ago

@Iomeroe After the Debian/Ubuntu issue with python-tornado only now recommending concurrent.futures, I checked the spec file for the Redhat / Centos builds and python-futures is a Requires:, so the packaging is correct. Installing by the bootstrap should have utilized the requirements file which also includes futures.

I shall take a closer look however and unit test to see if I can duplicate your issue, since I had thought the Redhat family should have been unaffected by this. In the past, I have seen issues with minimal CentOS builds where even sudo was not part of the minimal version of CentOS (RHEL 6), and the packages available in minimal installs changed between minor versions, for example: RHEL 6.1, 6.2, 6.3. Hence the problem may be due to the minimal version of CentOS not containing all of the required packages for Salt to function rather than a script or packaging error.

Can you provide the version of minimal CentOS that you are using, or confirm that it is CentOS Linux 7.5.1804 Core (from #50220)

dmurphy18 commented 5 years ago

@Iomeroe After installing CentOS 7.5 minimal, I found that python-futures is listed as a dependency for salt:

[root@localhost ~]# yum list python*-futures Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile

[root@localhost ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) [root@localhost ~]#

Installing salt-ssh also install python-futures Installed: salt-ssh.noarch 0:2018.3.3-1.el7

Dependency Installed: PyYAML.x86_64 0:3.11-1.el7 libsodium.x86_64 0:1.0.16-1.el7 libtomcrypt.x86_64 0:1.17-26.el7 libtommath.x86_64 0:0.42.0-6.el7
libxml2-python.x86_64 0:2.9.1-6.el7_2.3 libyaml.x86_64 0:0.1.4-11.el7_0 openpgm.x86_64 0:5.2.122-2.el7 pciutils.x86_64 0:3.5.1-3.el7
python-babel.noarch 0:0.9.6-8.el7 python-backports.x86_64 0:1.0-8.el7 python-backports-ssl_match_hostname.noarch 0:3.5.0.1-1.el7 python-chardet.noarch 0:2.2.1-1.el7_1
python-futures.noarch 0:3.0.3-1.el7 python-ipaddress.noarch 0:1.0.16-2.el7 python-jinja2.noarch 0:2.7.2-2.el7 python-kitchen.noarch 0:1.1.1-5.el7
python-markupsafe.x86_64 0:0.11-10.el7 python-msgpack.x86_64 0:0.4.6-1.el7 python-psutil.x86_64 0:2.2.1-1.el7 python-requests.noarch 0:2.6.0-1.el7_1
python-six.noarch 0:1.9.0-2.el7 python-tornado.x86_64 0:4.2.1-3.el7 python-urllib3.noarch 0:1.10.2-5.el7 python-zmq.x86_64 0:15.3.0-3.el7
python2-crypto.x86_64 0:2.6.1-15.el7 salt.noarch 0:2018.3.3-1.el7 systemd-python.x86_64 0:219-57.el7_5.3 yum-utils.noarch 0:1.1.31-46.el7_5
zeromq.x86_64 0:4.1.4-7.el7

dmurphy18 commented 5 years ago

The updated Debian 9 and Ubuntu 18.04 packages for 2017.7.8-2 and 2018.3.3-2 have been delivered to test.

dmurphy18 commented 5 years ago

@Iomeroe If the problem persists on Redhat, please open another issue and assign it to dmurphy18 and I shall look into it further. Want a different issue since this is dealing with Ubuntu/Debian.

Thanks

dmurphy18 commented 5 years ago

The fixes are now live on repo.saltstack.com

jryberg commented 5 years ago

Hi,

This problem exist on CentOS 7 as well but it does not seems this has been addressed? At least not for 2017.7.8

@dmurphy18, did you have another case for CentOS?

dmurphy18 commented 5 years ago

@jryberg A better place to comment would be salt-pack #598 since it is open and this issue is closed. But examining the current Salt for 2017.7.8, it would appear that python-futures is a dependency and is provided for Centos 7.

root@localhost:~# yum deplist salt Loaded plugins: fastestmirror, langpacks Loading mirror speeds from cached hostfile

maltris commented 5 years ago

Not sure why this is closed, but I think it also applies to the Ubuntu 16.04 repo, which as of today, does not contain +ds-2 and does not have the correct dependencies set.