[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails

Description

salt-ssh on a Python 3.9 host and a Python 3.6.15 target fails with ModuleNotFoundError: No module named '_contextvars' error:

Setup

My setup on the host is a Nix environment with Salt 3004 and a target machine with openSUSE and Python 3.9.15 on ARM, but from what I see in the code, it should happen on a Python > 3.7 host / Python 3.6 target.

Steps to Reproduce the behavior

Create a salt-ssh self-contained tree with a Saltfile, using a modern Python install.
Have a 3.6 target. You will need contextvars backport package (python3-contextvars)

After running salt-ssh rpi01 grains.items, it will fail:

        Traceback (most recent call last):
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.root_05dd7c_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/scripts.py", line 426, in salt_call
            import salt.cli.call
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/cli/caller.py", line 14, in <module>
            import salt.loader
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/__init__.py", line 14, in <module>
            import salt.config
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/config/__init__.py", line 101, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/config/__init__.py", line 89, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/grains/core.py", line 32, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/utils/templates.py", line 20, in <module>
            import salt.utils.jinja
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/utils/jinja.py", line 21, in <module>
            import salt.fileclient
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/fileclient.py", line 15, in <module>
            import salt.client
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/client/__init__.py", line 33, in <module>
            import salt.payload
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/payload.py", line 15, in <module>
            import salt.loader.context
          File "/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.root_05dd7c_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
    stdout:

I believe the reason could be (I could be wrong):

/var/tmp/.root_05dd7c_salt/pyall/salt/loader/context.py does:

try:
    # Try the stdlib C extension first
    import _contextvars as contextvars
except ImportError:
    # Py<3.7
    import contextvars

The fallback import should end importing contextvars backport. but because sys.path includes /var/tmp/.root_05dd7c_salt/py3/contextvars.py first, which is a copy of the 3.9 stdlib, which also depends on _contextvars!! native module:

from _contextvars import Context, ContextVar, Token, copy_context
__all__ = ('Context', 'ContextVar', 'Token', 'copy_context')

which brings Python 3.6 back into importing something that is not in Python 3.6, even if the backport package is installed on the machine.

So there are a few things I can't make sense of:

the fallback import with underscore / not underscore

The contextvars backport states:

The good news is that the standard library always takes the precedence over site packages, so even if a local contextvars module is installed, the one from the standard library will be used. Therefore you can simply list “contextvars” in your requirements.txt or setup.py files.

So I assume just importing just context in salt/loader/context.py should attempt to load the 3.7 module, or fallback to the backport.

contextvars being included in the thin!. The whole point of the backport is to implement contextvars without depending on _contextvars, so if we are on a newer Python, this module is already included, and if we are on < 3.6, the 3.7 module can't be used. What is the point of copying it?

Fixes and workarounds

Just deleting rm /var/tmp/.root_05dd7c_salt/py3/contextvars.py makes things work, as Python 3.6 would then use the installed contextvars backport.
I tried to use python 3.9 on the target machine. It would still use the "wrong" py3 dir module, but it would work as _contextvars is available. salt-ssh does not allow me to select /usr/bin/python3.9 as the interpreter to use in the target machine, so it uses /usr/bin/python3.

My intuition tells me there is no point in packing contextvars in the thin, unless we are packing the backport package.

Expected behavior

I expect salt-ssh rpi01 grains.items to work

salt --versions-report

Host: ``` Salt Version: Salt: 3004 Dependency Versions: cffi: 1.14.6 cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.0.2 libgit2: Not Installed M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.20 pycrypto: Not Installed pycryptodome: 3.11.0 pygit2: Not Installed Python: 3.9.6 (default, Jun 28 2021, 08:57:49) python-gnupg: Not Installed PyYAML: 5.4.1 PyZMQ: 21.0.2 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: opensuse-tumbleweed 20211215 n/a locale: utf-8 machine: x86_64 release: 5.15.8-1-default system: Linux version: openSUSE Tumbleweed 20211215 n/a ``` Target: ``` python3-3.6.13-3.81.2.aarch64 python3-contextvars-2.4-bp153.1.14.noarch ```

Duplicate of #59942? @dwoz there are other reports there that it wasn't fixed.

Also same issue, would love to see a workaround for this even if a real fix won't come for a good while

The issue seems to come from these lines: https://github.com/saltstack/salt/blob/38341cb2fe208498393780921a17cdf7963663c0/salt/utils/thin.py#L429-L435

contextvars is part of the standard library, so I'm not really understanding why this is packaging it if any python version from 3.7 up would already have it.

Given that a 3.7+ master is not going to have the backport (Or prefer it if its installed), this means that an incompatible version of contextvars gets packaged before being sent to the 3.6 target.

I was able to solve this in our system by commenting these lines, then using the pre-flight script to install the contextvars backport. Not sure right now how to best handle this without using pre-flight

Same here, commenting these lines on my salt-ssh client side (running with python 3.10), and installing contextvars using pip on my python3.6 server was successfull.

Python 3.10 contextvars.py is packaged in the salt-ssh thin but this contextvars.py version is not compatible if servers are running python 3.6/3.7 anyway.

This is already fixed, but is a bit counter-intuitive: you need to install contextvars backport package even when the salt-master is using Python 3.7+

An execution returning the already known error:

# salt-ssh 'ijurn.oci' -w -t test.ping
ijurn.oci:
    ----------
    retcode:   
        1
    stderr:
        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/scripts.py", line 435, in salt_call
            import salt.cli.call
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/caller.py", line 12, in <module>
            import salt.channel.client
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/channel/client.py", line 13, in <module>
            import salt.crypt
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/crypt.py", line 26, in <module>
            import salt.payload
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/payload.py", line 12, in <module>
            import salt.loader.context
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/__init__.py", line 15, in <module>
            import salt.config
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 100, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/grains/core.py", line 31, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/utils/templates.py", line 27, in <module>
            from salt.loader.context import NamedLoaderContext
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.opc_706a4b_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
        [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
        ModuleNotFoundError: No module named '_contextvars'
        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 10, in <module>
            import _contextvars as contextvars
        ModuleNotFoundError: No module named '_contextvars'

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/var/tmp/.opc_706a4b_salt/salt-call", line 27, in <module>
            salt_call()
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/scripts.py", line 435, in salt_call
            import salt.cli.call
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/call.py", line 3, in <module>
            import salt.cli.caller
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/cli/caller.py", line 12, in <module>
            import salt.channel.client
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/channel/client.py", line 13, in <module>
            import salt.crypt
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/crypt.py", line 26, in <module>
            import salt.payload
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/payload.py", line 12, in <module>
            import salt.loader.context
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/__init__.py", line 15, in <module>
            import salt.config
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 100, in <module>
            _DFLT_IPC_WBUFFER = _gather_buffer_space() * 0.5
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
            import salt.grains.core
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/grains/core.py", line 31, in <module>
            import salt.modules.cmdmod
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/modules/cmdmod.py", line 31, in <module>
            import salt.utils.templates
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/utils/templates.py", line 27, in <module>
            from salt.loader.context import NamedLoaderContext
          File "/var/tmp/.opc_706a4b_salt/pyall/salt/loader/context.py", line 13, in <module>
            import contextvars
          File "/var/tmp/.opc_706a4b_salt/py3/contextvars.py", line 1, in <module>
            from _contextvars import Context, ContextVar, Token, copy_context
        ModuleNotFoundError: No module named '_contextvars'
    stdout:

Then, install contextvars and run the same command:

# pip install contextvars
Collecting contextvars
  Using cached contextvars-2.4-py3-none-any.whl
Requirement already satisfied: immutables>=0.9 in /usr/lib/python3.9/site-packages (from contextvars) (0.18)
Installing collected packages: contextvars
Successfully installed contextvars-2.4
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
# salt-ssh 'ijurn.oci' -w -t test.ping
ijurn.oci:
    True

In the requires.txt, even when using Python 3.7+, the contextvars module is listed as a requirement:

Jinja2
msgpack>=0.5,!=0.5.5
PyYAML
MarkupSafe
requests>=1.0.0
distro>=1.0.1
contextvars
psutil>=5.0.0
pyzmq>19.0.2 ; python_version >= "3.9"
pycryptodomex>=3.9.8

And it's there exactly to make salt-ssh works OK with targets using versions of Python lower than 3.7, as @bryceml stated in #60483:

if downstream packages want to remove contextvars from the requirements on py3.7+, that's up to them, it just won't work with salt-ssh targets less than py3.7.

Take care that the thin package should be regenerated after contextvars installation.

Thanks for the detailed walkthrough @piterpunk.

I've followed your instructions and can confirms that on my setup (Archlinux source salt-ssh on a Python 3.9 virtualenv, Rockylinux 8 target using the system Python 3.6), installing contextvars in the source virtualenv then re-generating the thin package does not seem to be sufficient to work the issue around.

This is particularly confusing considering contextvars==2.4 is already installed on the target host, as part of a regular salt-3004 on python3.6 install.

salt --versions-report

```yaml Salt Version: Salt: 3004.2 Dependency Versions: cffi: 1.15.0 cherrypy: Not Installed dateutil: Not Installed docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 3.1.2 libgit2: 1.1.0 M2Crypto: Not Installed Mako: Not Installed msgpack: 1.0.4 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: 2.21 pycrypto: Not Installed pycryptodome: 3.15.0 pygit2: 1.5.0 Python: 3.9.12 (main, May 15 2022, 19:24:24) python-gnupg: Not Installed PyYAML: 6.0 PyZMQ: 21.0.2 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.3 System Versions: dist: arch locale: utf-8 machine: x86_64 release: 5.18.6-arch1-1 system: Linux version: Arch Linux ``` salt-call --versions-report (on target host) ```yaml Salt Version: Salt: 3004.2 Dependency Versions: cffi: Not Installed cherrypy: Not Installed dateutil: 2.6.1 docker-py: Not Installed gitdb: Not Installed gitpython: Not Installed Jinja2: 2.10.1 libgit2: Not Installed M2Crypto: 0.35.2 Mako: Not Installed msgpack: 0.6.2 msgpack-pure: Not Installed mysql-python: Not Installed pycparser: Not Installed pycrypto: Not Installed pycryptodome: Not Installed pygit2: Not Installed Python: 3.6.8 (default, Apr 12 2022, 06:55:39) python-gnupg: Not Installed PyYAML: 3.12 PyZMQ: 19.0.0 smmap: Not Installed timelib: Not Installed Tornado: 4.5.3 ZMQ: 4.3.4 System Versions: dist: rocky 8.6 Green Obsidian locale: UTF-8 machine: x86_64 release: 4.18.0-372.9.1.el8.x86_64 system: Linux version: Rocky Linux 8.6 Green Obsidian ```

EDIT: Turns out I've stumbled upon this issue already back in January and I found another workaround on a duplicate issue, https://github.com/saltstack/salt/issues/59942#issuecomment-1020494839 that's related to an issue with typing_extensions. My memory is poor !

To whom it may concern, if Piter's instructions aren't sufficient because your issue is caused by typing_extensions, install typing_extensions (version 4.1.1 maximum, 4.2.0 introduces a breaking change...), then follow those instructions https://github.com/saltstack/salt/issues/59942#issuecomment-961153192

I just installed salt-ssh on a centos 7 via rpm repo. When connecting to another centos 7 machine, after installing python3.x86_64 and python36-distro.noarch (because got another error as well), got this error:

salt-ssh -v -i '*' test.ping Executing job with jid 20221216202157353872

vm34:

retcode:
    1
stderr:
    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 10, in <module>
        import _contextvars as contextvars
    ModuleNotFoundError: No module named '_contextvars'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/salt-call", line 27, in <module>
        salt_call()
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/scripts.py", line 435, in salt_call
        import salt.cli.call
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/call.py", line 3, in <module>
        import salt.cli.caller
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/caller.py", line 12, in <module>
        import salt.channel.client
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/channel/client.py", line 13, in <module>
        import salt.crypt
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/crypt.py", line 26, in <module>
        import salt.payload
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/payload.py", line 12, in <module>
        import salt.loader.context
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/__init__.py", line 15, in <module>
        import salt.config
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 100, in <module>
        _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
        import salt.grains.core
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/grains/core.py", line 31, in <module>
        import salt.modules.cmdmod
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
        import salt.utils.templates
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/utils/templates.py", line 27, in <module>
        from salt.loader.context import NamedLoaderContext
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 13, in <module>
        import contextvars
    ModuleNotFoundError: No module named 'contextvars'
    [ERROR   ] An un-handled exception was caught by Salt's global exception handler:
    ModuleNotFoundError: No module named 'contextvars'
    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 10, in <module>
        import _contextvars as contextvars
    ModuleNotFoundError: No module named '_contextvars'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/var/tmp/.saltuser_44a047_salt/salt-call", line 27, in <module>
        salt_call()
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/scripts.py", line 435, in salt_call
        import salt.cli.call
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/call.py", line 3, in <module>
        import salt.cli.caller
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/cli/caller.py", line 12, in <module>
        import salt.channel.client
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/channel/client.py", line 13, in <module>
        import salt.crypt
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/crypt.py", line 26, in <module>
        import salt.payload
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/payload.py", line 12, in <module>
        import salt.loader.context
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/__init__.py", line 15, in <module>
        import salt.config
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 100, in <module>
        _DFLT_IPC_WBUFFER = int(_gather_buffer_space() * 0.5)
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/config/__init__.py", line 88, in _gather_buffer_space
        import salt.grains.core
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/grains/core.py", line 31, in <module>
        import salt.modules.cmdmod
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/modules/cmdmod.py", line 32, in <module>
        import salt.utils.templates
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/utils/templates.py", line 27, in <module>
        from salt.loader.context import NamedLoaderContext
      File "/var/tmp/.saltuser_44a047_salt/pyall/salt/loader/context.py", line 13, in <module>
        import contextvars
    ModuleNotFoundError: No module named 'contextvars'
stdout:

EDIT: OK, doing pip3 install contextvars fixed it

saltstack / salt

[BUG] salt-ssh from newer Python (3.9) to 3.6.x host fails #61419

salt-ssh -v -i '*' test.ping Executing job with jid 20221216202157353872

vm34: