{CentOS7} 2016.3.2: Targeting more than 1 minion with cmd.run results in an error " Passed invalid arguments to cmd.run: __init__() got an unexpected keyword argument 'bg"

[StoyanV]

Description of Issue/Question

when targeting more than one minion using the cmd.run option, I get error: "Passed invalid arguments to cmd.run: init() got an unexpected keyword argument 'bg" The command however completes just fine, only I get this very annoying output appended after the command is executed.

Working commands: -----------Example1-----------

[root]$ salt '*' test.ping
messaging-01. :
    True
cloudcontrol-01. :
    True
bastion-01. :
    True
app-01. :
    True
web-01. :
    True

-----------END1----------- -----------Example2-----------

[root]$ salt 'app*' cmd.run 'echo "12356"'
app-01.

-----------END2----------- -----------Example3-----------

[root]$ salt -C 'app* or web*' test.versions
app-01.:
    Salt Version:
               Salt: 2016.3.2
#output truncated

web-01.:
    Salt Version:
               Salt: 2016.3.2
#output truncated

-----------END3-----------

BUT:

    [root]$ salt -C 'app* or web*' cmd.run 'echo 123456'
web-01. :
    123456
app-01. :
    123456
web-01. :
    Passed invalid arguments to cmd.run: __init__() got an unexpected keyword argument 'bg'

        Execute the passed command and return the output as a string

        Note that ``env`` represents the environment variables for the command, and
        should be formatted as a dict, or a YAML string which resolves to a dict.

        :param str cmd: The command to run. ex: ``ls -lart /home``

        :param str cwd: The current working directory to execute the command in,
          defaults to ``/root`` (``C:\`` in windows)

        :param str stdin: A string of standard input can be specified for the
          command to be run using the ``stdin`` parameter. This can be useful in cases
          where sensitive information must be read from standard input.:

        :param str runas: User to run script as. If running on a Windows minion you
          must also pass a password

        :param str password: Windows only. Pass a password if you specify runas.
          This parameter will be ignored for other OS's

          .. versionadded:: 2016.3.0

        :param str shell: Shell to execute under. Defaults to the system default
          shell.

        :param bool python_shell: If False, let python handle the positional
          arguments. Set to True to use shell features, such as pipes or redirection

        :param bool bg: If True, run command in background and do not await or deliver it's results

        :param list env: A list of environment variables to be set prior to
          execution.

            Example:

            .. code-block:: yaml

                salt://scripts/foo.sh:
                  cmd.script:
                    - env:
                      - BATCH: 'yes'

            .. warning::

                The above illustrates a common PyYAML pitfall, that **yes**,
                **no**, **on**, **off**, **true**, and **false** are all loaded as
                boolean ``True`` and ``False`` values, and must be enclosed in
                quotes to be used as strings. More info on this (and other) PyYAML
                idiosyncrasies can be found :doc:`here
                </topics/troubleshooting/yaml_idiosyncrasies>`.

            Variables as values are not evaluated. So $PATH in the following
            example is a literal '$PATH':

            .. code-block:: yaml

                salt://scripts/bar.sh:
                  cmd.script:
                    - env: "PATH=/some/path:$PATH"

            One can still use the existing $PATH by using a bit of Jinja:

            .. code-block:: yaml

                {% set current_path = salt['environ.get']('PATH', '/bin:/usr/bin') %}

                mycommand:
                  cmd.run:
                    - name: ls -l /
                    - env:
                      - PATH: {{ [current_path, '/my/special/bin']|join(':') }}

        :param bool clean_env: Attempt to clean out all other shell environment
          variables and set only those provided in the 'env' argument to this
          function.

        :param str template: If this setting is applied then the named templating
          engine will be used to render the downloaded file. Currently jinja, mako,
          and wempy are supported

        :param bool rstrip: Strip all whitespace off the end of output before it is
          returned.

        :param str umask: The umask (in octal) to use when running the command.

        :param str output_loglevel: Control the loglevel at which the output from
          the command is logged. Note that the command being run will still be logged
          (loglevel: DEBUG) regardless, unless ``quiet`` is used for this value.

        :param int timeout: A timeout in seconds for the executed process to return.

        :param bool use_vt: Use VT utils (saltstack) to stream the command output
          more interactively to the console and the logs. This is experimental.

        .. warning::
            This function does not process commands through a shell
            unless the python_shell flag is set to True. This means that any
            shell-specific functionality such as 'echo' or the use of pipes,
            redirection or &&, should either be migrated to cmd.shell or
            have the python_shell=True flag set here.

            The use of python_shell=True means that the shell will accept _any_ input
            including potentially malicious commands such as 'good_command;rm -rf /'.
            Be absolutely certain that you have sanitized your input prior to using
            python_shell=True

        CLI Example:

        .. code-block:: bash

            salt '*' cmd.run "ls -l | awk '/foo/{print \\$2}'"

        The template arg can be set to 'jinja' or another supported template
        engine to render the command arguments before execution.
        For example:

        .. code-block:: bash

            salt '*' cmd.run template=jinja "ls -l /tmp/{{grains.id}} | awk '/foo/{print \\$2}'"

        Specify an alternate shell with the shell parameter:

        .. code-block:: bash

            salt '*' cmd.run "Get-ChildItem C:\\ " shell='powershell'

        A string of standard input can be specified for the command to be run using
        the ``stdin`` parameter. This can be useful in cases where sensitive
        information must be read from standard input.:

        .. code-block:: bash

            salt '*' cmd.run "grep f" stdin='one\\ntwo\\nthree\\nfour\\nfive\\n'

        If an equal sign (``=``) appears in an argument to a Salt command it is
        interpreted as a keyword argument in the format ``key=val``. That
        processing can be bypassed in order to pass an equal sign through to the
        remote shell command by manually specifying the kwarg:

        .. code-block:: bash

            salt '*' cmd.run cmd='sed -e s/=/:/g'

app-01. :
    Passed invalid arguments to cmd.run: __init__() got an unexpected keyword argument 'bg'

        Execute the passed command and return the output as a string

        Note that ``env`` represents the environment variables for the command, and
        should be formatted as a dict, or a YAML string which resolves to a dict.

        :param str cmd: The command to run. ex: ``ls -lart /home``

        :param str cwd: The current working directory to execute the command in,
          defaults to ``/root`` (``C:\`` in windows)

        :param str stdin: A string of standard input can be specified for the
          command to be run using the ``stdin`` parameter. This can be useful in cases
          where sensitive information must be read from standard input.:

        :param str runas: User to run script as. If running on a Windows minion you
          must also pass a password

        :param str password: Windows only. Pass a password if you specify runas.
          This parameter will be ignored for other OS's

          .. versionadded:: 2016.3.0

        :param str shell: Shell to execute under. Defaults to the system default
          shell.

        :param bool python_shell: If False, let python handle the positional
          arguments. Set to True to use shell features, such as pipes or redirection

        :param bool bg: If True, run command in background and do not await or deliver it's results

        :param list env: A list of environment variables to be set prior to
          execution.

            Example:

            .. code-block:: yaml

                salt://scripts/foo.sh:
                  cmd.script:
                    - env:
                      - BATCH: 'yes'

            .. warning::

                The above illustrates a common PyYAML pitfall, that **yes**,
                **no**, **on**, **off**, **true**, and **false** are all loaded as
                boolean ``True`` and ``False`` values, and must be enclosed in
                quotes to be used as strings. More info on this (and other) PyYAML
                idiosyncrasies can be found :doc:`here
                </topics/troubleshooting/yaml_idiosyncrasies>`.

            Variables as values are not evaluated. So $PATH in the following
            example is a literal '$PATH':

            .. code-block:: yaml

                salt://scripts/bar.sh:
                  cmd.script:
                    - env: "PATH=/some/path:$PATH"

            One can still use the existing $PATH by using a bit of Jinja:

            .. code-block:: yaml

                {% set current_path = salt['environ.get']('PATH', '/bin:/usr/bin') %}

                mycommand:
                  cmd.run:
                    - name: ls -l /
                    - env:
                      - PATH: {{ [current_path, '/my/special/bin']|join(':') }}

        :param bool clean_env: Attempt to clean out all other shell environment
          variables and set only those provided in the 'env' argument to this
          function.

        :param str template: If this setting is applied then the named templating
          engine will be used to render the downloaded file. Currently jinja, mako,
          and wempy are supported

        :param bool rstrip: Strip all whitespace off the end of output before it is
          returned.

        :param str umask: The umask (in octal) to use when running the command.

        :param str output_loglevel: Control the loglevel at which the output from
          the command is logged. Note that the command being run will still be logged
          (loglevel: DEBUG) regardless, unless ``quiet`` is used for this value.

        :param int timeout: A timeout in seconds for the executed process to return.

        :param bool use_vt: Use VT utils (saltstack) to stream the command output
          more interactively to the console and the logs. This is experimental.

        .. warning::
            This function does not process commands through a shell
            unless the python_shell flag is set to True. This means that any
            shell-specific functionality such as 'echo' or the use of pipes,
            redirection or &&, should either be migrated to cmd.shell or
            have the python_shell=True flag set here.

            The use of python_shell=True means that the shell will accept _any_ input
            including potentially malicious commands such as 'good_command;rm -rf /'.
            Be absolutely certain that you have sanitized your input prior to using
            python_shell=True

        CLI Example:

        .. code-block:: bash

            salt '*' cmd.run "ls -l | awk '/foo/{print \\$2}'"

        The template arg can be set to 'jinja' or another supported template
        engine to render the command arguments before execution.
        For example:

        .. code-block:: bash

            salt '*' cmd.run template=jinja "ls -l /tmp/{{grains.id}} | awk '/foo/{print \\$2}'"

        Specify an alternate shell with the shell parameter:

        .. code-block:: bash

            salt '*' cmd.run "Get-ChildItem C:\\ " shell='powershell'

        A string of standard input can be specified for the command to be run using
        the ``stdin`` parameter. This can be useful in cases where sensitive
        information must be read from standard input.:

        .. code-block:: bash

            salt '*' cmd.run "grep f" stdin='one\\ntwo\\nthree\\nfour\\nfive\\n'

        If an equal sign (``=``) appears in an argument to a Salt command it is
        interpreted as a keyword argument in the format ``key=val``. That
        processing can be bypassed in order to pass an equal sign through to the
        remote shell command by manually specifying the kwarg:

        .. code-block:: bash

            salt '*' cmd.run cmd='sed -e s/=/:/g'

Setup

https://repo.saltstack.com/#rhel followed this verbatim...

Steps to Reproduce Issue

Install salt from official documentation. Setup simple master-minion config Restart salt-minion twice as indicated in different open bugs

Versions Report

Salt Version:
           Salt: 2016.3.2

Dependency Versions:
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.5.3
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.21.0
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: 0.21.4
         Python: 2.7.5 (default, Nov 20 2015, 02:00:19)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.2.1511 Core
        machine: x86_64
        release: 3.10.0-327.28.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.2.1511 Cor

[StoyanV]

Just for the information - running a highstate on the individual machines proved not to work either. So in the end it turned out to be this: Post install (which in my case was an upgrade from 2015.08.08) there were a lot of "dead" salt-minions running in the background, which did not get stopped by systemctl restart salt-minion. I needed to kill them all manually and restart the process afterwards. Now all works as expected.

[Ch3LL]

@StoyanV i think what you might have run into is this issue #33516 Are you okay closing this issue since its resolved and there is already an issue open for the other issue you found during troubleshooting?

[StoyanV]

Definately.