Problem to run command on external host (Cloud)

[FelipoAntonoff]

Describe the bug

Problem to run command on external host Normal runs if @local, but for the Cloud the error

To Reproduce

Example command: pyinfra /home/vagrant/pyinfra/inventory.py --limit 159.89.243.96 --debug exec -- echo "hello world"

Expected behavior

It was to return like @local: Beginning operation run... --> Starting operation: Server/Shell (echo hello world) [pyinfra.api.operations] Starting operation Server/Shell on @local [pyinfra.api.connectors.local] --> Running command on localhost: sh -c 'echo hello world' [@local] hello world

Meta

Support information Local run Vagrant port open:

    System: Linux
      Platform: Linux-5.10.0-9-amd64-x86_64-with-glibc2.31
      Release: 5.10.0-9-amd64
      Machine: x86_64
    pyinfra: v1.5
    Executable: /usr/local/bin/pyinfra
    Python: 3.9.2 (CPython, GCC 10.2.1 20210110)

Support information Cloud 159.89.243...

  System: Linux
      Platform: Linux-5.10.0-10-amd64-x86_64-with-glibc2.17
      Release: 5.10.0-10-amd64
      Machine: x86_64
    pyinfra: v1.5
    Executable: /usr/local/bin/pyinfra
    Python: 3.8.2 (CPython, GCC 4.9.2)

Install by pip.

Return debug:

[159.89.243...] Connected
    [pyinfra.api.state] Activating host: 159.89.243..
--> An unexpected internal exception occurred:

  File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation_kwargs.py", line 59, in generate_env
    env.update(value)
ValueError: dictionary update sequence element #0 has length 1; 2 is required
    [pyinfra_cli.exceptions]   File "/usr/local/lib/python3.9/dist-packages/pyinfra_cli/main.py", line 236, in cli
    _main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyinfra_cli/main.py", line 594, in _main
    add_op(
  File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation.py", line 95, in add_op
    results[host] = op_func(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation.py", line 183, in decorated_func
    global_kwargs, global_kwarg_keys = pop_global_op_kwargs(state, host, kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation_kwargs.py", line 171, in pop_global_op_kwargs
    value = handler(state.config, value)
  File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation_kwargs.py", line 59, in generate_env
    env.update(value)

    [pyinfra_cli.exceptions] ValueError: dictionary update sequence element #0 has length 1; 2 is required

--> The full traceback has been written to pyinfra-debug.log
--> If this is unexpected please consider submitting a bug report on GitHub, for more information run `pyinfra --support`.

If I also run on the test.py file:

apt.packages(
     name='Install Docker',
     packages=['podman'],
     present=True
)

Returns the error:

   File "/usr/local/lib/python3.9/dist-packages/pyinfra_cli/util.py", line 80, in exec_file
     exec(PYTHON_CODES[filename], date)
   File "/home/vagrant/pyinfra/teste.py", line 16, in <module>
     apt.packages(
   File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation.py", line 183, in decorated_func
     global_kwargs, global_kwarg_keys = pop_global_op_kwargs(state, host, kwargs)
   File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation_kwargs.py", line 171, in pop_global_op_kwargs
     value = handler(state.config, value)
   File "/usr/local/lib/python3.9/dist-packages/pyinfra/api/operation_kwargs.py", line 59, in generate_env
     env.update(value)
ValueError: dictionary update sequence element #0 has length 1; 2 is required

@local catches normally Already the lsb_release = host.fact.lsb_release gets in the Cloud too

Content File teste.py:

from pyinfra import config, host, logger
from pyinfra.operations import apt, server, files, pip
from pyinfra.facts.files import File
from pyinfra.facts.server import *

import json
import sys

config.SUDO = True
# If the remote system requires a password for sudo, un-comment the line below:
# USE_SUDO_PASSWORD = True

lsb_release = host.fact.lsb_release
#sys.stdout.write(lsb_release['id'])

apt.packages(
    name='Instalar Podman',
    packages=['podman'],
    present=True
)

Run: pyinfra /home/vagrant/pyinfra/inventory.py --limit 159.89.243.96 /home/vagrant/pyinfra/teste.py -vv --debugg ERROR Connection OK

Local: pyinfra /home/vagrant/pyinfra/inventory.py --limit @local /home/vagrant/pyinfra/teste.py -vv --debug OK

[Fizzadar]

Hi @FelipoAntonoff would you mind posting the inventory file here? I’m specifically interested in any env data or config that could be causing this.

The error here looks like a list is being passed as env instead of a dictionary but can’t be certain.

[FelipoAntonoff]

Hello @Fizzadar , here is the inventory used:

# https://github.com/Fizzadar/pyinfra/blob/current/examples/inventory.py https://docs.pyinfra.com/en/1.x/cli.html#inventory
local = [
    # Vagrant Local ou VM
    '@local'
]

dev = [
    # Digital Ocean Local
    ('159.89.243.96', 
        {
            'name': 'Digital Ocean Dev',
            'ssh_port': '2222',
            'ssh_user': 'root',
            'ssh_key': '/etc/ssh_vagrant/DigitalOcean122021.key',
            'ssh_key_password': '....'
        }
    )
]

From what I can see, it connects the machine, but when it starts running, for example, an apt or another resource, the error appears.

Locally in Vagrant I keep running some scripts and all of them getting normal if I use @local .

[edmond-edited]

Not sure if this is too late, but I encountered the same problem.

Turns out that the this only happens from v1.5 onwards. For me it was because previously host.data.* wasn't considered for global arguments (https://github.com/Fizzadar/pyinfra/releases/tag/v1.5) and we were using host.data.env as a string instead of a dictionary. Ultimately, when you try to update a dict with a string that's the error that comes up!

[Fizzadar]

@edmond-edited YES! This suddenly makes sense, a side effect I had not considered. I've applied a workaround in https://github.com/Fizzadar/pyinfra/commit/1c2b0edd3c8c51fafd576e91482ff204907349c1 and released that in v1.6.2. @FelipoAntonoff does this also fix your issue?

Long term the plan is now to prefix all the global arguments with _ which will prevent such issues.

[Fizzadar]

v2 & _ prefixed global arguments are now live :)

[FelipoAntonoff]

Hi @Fizzadar , thank you very much.

I ran the new version and it's getting 100% even for external host.

Solved my problem. Sorry for the delay in testing.

I ran a test with server.shell, files.get, apt.packages and others and they all started normally on the Cloud, even very quickly as if I had it in the Cloud terminal, detail I'm running Pyinfra in São Paulo/Brazil and Cloud is in the USA.

Thanks.

[Fizzadar]

@FelipoAntonoff amazing! Thank you for following up, much apprecaited :)