search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.22k stars 5.49k forks source link

salt-ssh not working on remote hosts with python versions >3.8 #61276

Open HappyBuzzCut opened 3 years ago

HappyBuzzCut commented 3 years ago

Description of Issue

When trying to highstate a new Fedora 25 machine with Python 3.11 (currently latest in repo), I get the following error: "ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/init.py)"

Setup

N/A - Errors with salt using Python module. I have no custom python scripts

Steps to Reproduce Issue

[root@morton ~]
# salt-ssh -i comms state.highstate
comms:
    ----------
    _error:
        Failed to return clean data
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_5a678c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_5a678c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/caller.py", line 15, in <module>
            import salt.minion
          File "/var/tmp/.root_5a678c_salt/pyall/salt/minion.py", line 23, in <module>
            import salt.client
          File "/var/tmp/.root_5a678c_salt/pyall/salt/client/__init__.py", line 40, in <module>
            import salt.utils.minions
          File "/var/tmp/.root_5a678c_salt/pyall/salt/utils/minions.py", line 12, in <module>
            import salt.auth.ldap
          File "/var/tmp/.root_5a678c_salt/pyall/salt/auth/ldap.py", line 11, in <module>
            from jinja2 import Environment
          File "/var/tmp/.root_5a678c_salt/pyall/jinja2/__init__.py", line 6, in <module>
            from markupsafe import escape
          File "/var/tmp/.root_5a678c_salt/py3/markupsafe/__init__.py", line 13, in <module>
            from collections import Mapping
        ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)
    stdout:

Versions Report

minion: None - Using salt-ssh to highstate a host over ssh. Remote host running fedora 35

master:

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.1
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Nov 18 2021, 10:07:16)
  python-gnupg: Not Installed
        PyYAML: 3.13
         PyZMQ: 17.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: oracle 7.9
        locale: UTF-8
       machine: x86_64
       release: 5.4.17-2136.300.7.el7uek.x86_64
        system: Linux
       version: Oracle Linux Server 7.9
welcome[bot] commented 3 years ago

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

OrangeDog commented 3 years ago

What version of markupsafe is that? It may simply need updating.

HappyBuzzCut commented 3 years ago

What version of markupsafe is that? It may simply need updating.

So in an effort to get this working, I have downgraded salt-ssh to version 3003.3 but the issue remains. I have made sure the package you mentioned was upgraded and I even installed it on remote machine. Here are the versions

salt master

Name         : python36-markupsafe
Version      : 0.23
Release      : 4.el7
Arch         : x86_64
Size         : 99 k
Source       : python3-markupsafe-0.23-4.el7.src.rpm
Repo         : @System
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python
URL          : http://pypi.python.org/pypi/MarkupSafe
License      : BSD
Description  : A library for safe markup escaping.

fedora 35 machine

Installed Packages
Name         : python3-markupsafe
Version      : 2.0.0
Release      : 2.fc35
Architecture : x86_64
Size         : 48 k
Source       : python-markupsafe-2.0.0-2.fc35.src.rpm
Repository   : @System
From repo    : fedora
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python 3
URL          : https://pypi.org/project/MarkupSafe/
License      : BSD
Description  : A library for safe markup escaping. Python 3 version.
OrangeDog commented 3 years ago

from collections import Mapping was removed in version 1.1 of markupsafe. Therefore it's the version from the master that its is using, which is far too old.

Salt-SSH should be using the packages it finds on the minion though, if they're installed. You say you're trying to run this under Python 3.11, but the stacktrace says /usr/lib64/python3.10/. Have you got the PATH/set_path configured correctly?

HappyBuzzCut commented 3 years ago

from collections import Mapping was removed in version 1.1 of markupsafe. Therefore it's the version from the master that its is using, which is far too old.

Okay the Salt folks need to get this package upgraded on their repo because as of now version 0.23-4 is in their rhel7 repo for versions 3003 and 3004

Salt-SSH should be using the packages it finds on the minion though, if they're installed. You say you're trying to run this under Python 3.11, but the stacktrace says /usr/lib64/python3.10/. Have you got the PATH/set_path configured correctly?

It is the same error for both the versions. 3.10 is the default and I just symlink /usr/bin/python3 to the ver I want. This of course is all in vain since like you said the master uses an older markupsafe

Interesting to note, my internal fedora 35 machine works just great with salt-minion but salt-ssh is problematic it seems with these mismatches

OrangeDog commented 3 years ago

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either: a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

HappyBuzzCut commented 3 years ago

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either: a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

'a' can be ruled out:

# dnf info python3-markupsafe.x86_64
Last metadata expiration check: 2:28:48 ago on Wed 24 Nov 2021 06:15:06 PM CST.
Installed Packages
Name         : python3-markupsafe
Version      : 2.0.0
Release      : 2.fc35
Architecture : x86_64
Size         : 48 k
Source       : python-markupsafe-2.0.0-2.fc35.src.rpm
Repository   : @System
From repo    : fedora
Summary      : Implements a XML/HTML/XHTML Markup safe string for Python 3
URL          : https://pypi.org/project/MarkupSafe/
License      : BSD
Description  : A library for safe markup escaping. Python 3 version.
# rpm -ql python3-markupsafe
/usr/lib/.build-id
/usr/lib/.build-id/8d
/usr/lib/.build-id/8d/29e1de80aecbb909f342cd7e34ea212b188fd8
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/PKG-INFO
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/SOURCES.txt
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/dependency_links.txt
/usr/lib64/python3.10/site-packages/MarkupSafe-2.0.0-py3.10.egg-info/top_level.txt
/usr/lib64/python3.10/site-packages/markupsafe
/usr/lib64/python3.10/site-packages/markupsafe/__init__.py
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/__init__.cpython-310.opt-1.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/__init__.cpython-310.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/_native.cpython-310.opt-1.pyc
/usr/lib64/python3.10/site-packages/markupsafe/__pycache__/_native.cpython-310.pyc
/usr/lib64/python3.10/site-packages/markupsafe/_native.py
/usr/lib64/python3.10/site-packages/markupsafe/_speedups.cpython-310-x86_64-linux-gnu.so
/usr/lib64/python3.10/site-packages/markupsafe/_speedups.pyi
/usr/lib64/python3.10/site-packages/markupsafe/py.typed
/usr/share/doc/python3-markupsafe
/usr/share/doc/python3-markupsafe/CHANGES.rst
/usr/share/doc/python3-markupsafe/README.rst
/usr/share/licenses/python3-markupsafe
/usr/share/licenses/python3-markupsafe/LICENSE.rst
# python3 -V
Python 3.10.0

Pip had markupsafe 2.0.1 installed. I even tried removing that to strictly use the rpm version. still failed...

# pip3.10 list|grep MarkupSafe
MarkupSafe         2.0.0
OrangeDog commented 3 years ago

'a' cannot be ruled out until you establish which python installation the minion is using.

Have you got the PATH/set_path configured correctly?

The python binary which that uses should include /usr/lib64/python3.10/site-packages in python -c 'import sys; print(sys.path)'

The original stacktrace had /usr/lib64/python3.10, but you've been changing things since then.

HappyBuzzCut commented 3 years ago

'a' cannot be ruled out until you establish which python installation the minion is using.

Have you got the PATH/set_path configured correctly?

The python binary which that uses should include /usr/lib64/python3.10/site-packages in python -c 'import sys; print(sys.path)'

# salt-ssh -i comms -r "python3 -c 'import sys; print(sys.path)'"
comms:
    ----------
    retcode:
        0
    stderr:
    stdout:
        ['', '/usr/lib64/python310.zip', '/usr/lib64/python3.10', '/usr/lib64/python3.10/lib-dynload', '/usr/local/lib64/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages', '/usr/local/lib/python3.10/site-packages/cloud_init-21.3-py3.10.egg', '/usr/lib/python3.10/site-packages', '/usr/lib64/python3.10/site-packages']
# salt-ssh -i comms -r "ls /usr/lib64/python3.10/site-packages|grep MarkupSafe"
comms:
    ----------
    retcode:
        0
    stderr:
    stdout:
        MarkupSafe-2.0.0-py3.10.egg-info

The original stacktrace had /usr/lib64/python3.10, but you've been changing things since then.

Nothing has changed.....

# salt-ssh -i comms test.ping
comms:
    ----------
    retcode:
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.root_5a678c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_5a678c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_5a678c_salt/pyall/salt/cli/caller.py", line 15, in <module>
            import salt.minion
          File "/var/tmp/.root_5a678c_salt/pyall/salt/minion.py", line 23, in <module>
            import salt.client
          File "/var/tmp/.root_5a678c_salt/pyall/salt/client/__init__.py", line 40, in <module>
            import salt.utils.minions
          File "/var/tmp/.root_5a678c_salt/pyall/salt/utils/minions.py", line 12, in <module>
            import salt.auth.ldap
          File "/var/tmp/.root_5a678c_salt/pyall/salt/auth/ldap.py", line 11, in <module>
            from jinja2 import Environment
          File "/var/tmp/.root_5a678c_salt/pyall/jinja2/__init__.py", line 6, in <module>
            from markupsafe import escape
          File "/var/tmp/.root_5a678c_salt/py3/markupsafe/__init__.py", line 13, in <module>
            from collections import Mapping
        ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib64/python3.10/collections/__init__.py)
    stdout:
OrangeDog commented 3 years ago

Yep, that looks like it's not picking up installed packages corectly then.

Installing a newer markupsafe (e.g. via yum or pip) on the master should work around it.

ESNewmanium commented 1 year ago

@HappyBuzzCut I was able to sidestep this by pip installing MarkupSafe 2.0.0 on my salt-ssh master which runs py 3.6. Do be sure to clear out the /var/cache/salt/master files of your salt-ssh master though. Didn't work until I did that.

miukumac commented 1 year ago

This problem is still open at least on a mixed openSUSE setup where the Master has Python 3.6.15 with Salt Master 3006.0 and some of the nodes with openSUSE Tumbleweed running on Python 3.11.4 and naturally on Minion 3006.0.

Installing a newer markupsafe did not triage the issue or perhaps it's because of the distribution packages.

OrangeDog commented 1 year ago

did not triage the issue

"triage" means to determine the severity and priority of something

miukumac commented 1 year ago

"triage" means to determine the severity and priority of something

Sorry, English isn't my primary language so sometimes I have to rely on translators and the verb we use for fixing translated to triage for some reason.

Anyway unfortunately upgrading the python module did not fix the issue for me.

matwey commented 1 year ago

The master is fine to be using an older markupsafe. There's nothing wrong there; it's the correct package for el7.

The problem is either: a. You have not installed markupsafe correctly on the minion for the python that salt-ssh is configured to use b. There is a bug in salt-ssh where it's not picking up dependencies already available on the minion

Do I understand correctly, that salt-ssh is able to use python dependencies from the minion side instead of providing it within the tarball?

Given you've been switching around with dev versions of Python, my guess would be a. If you can rule that out then it is b.

matwey commented 1 year ago

@OrangeDog from what I see from the sources, markupsafe is unconditionally added to the thin tarball. salt-call just prepends sys.path with bundled paths, so the target python has no chance to use installed markupsafe.