Salt-cloud Vmware parallel deployments fail with UnicodeDecodeError

javaria105 commented 4 years ago

Description of Issue

When deploying VMs with Salt-cloud in parallel, we get an error message that says "'utf8' codec can't decode byte 0xe2 in position 1023". The command we use is salt-cloud -P -m <map> -ldebug, with "map" being a yaml map file. The full error message is below:

[ERROR   ] Caught Exception, terminating workers
TRACE: 'utf8' codec can't decode byte 0xe2 in position 1023: unexpected end of data
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 67, in _call
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 2294, in create_multiprocessing
    local_master=parallel_data['local_master']
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1253, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2936, in create
    out = __utils__['cloud.bootstrap'](vm_, __opts__)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 644, in bootstrap
    deployed = deploy_script(**deploy_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 1762, in deploy_script
    if root_cmd(deploy_command, tty, sudo, **ssh_kwargs) != 0:
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2367, in root_cmd
    retcode = _exec_ssh_cmd(cmd, allow_failure=allow_failure, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2016, in _exec_ssh_cmd
    stdout, stderr = proc.recv()
  File "/usr/lib/python2.7/site-packages/salt/utils/vt.py", line 311, in recv
    return self._recv(maxsize)
  File "/usr/lib/python2.7/site-packages/salt/utils/vt.py", line 680, in _recv
    os.read(self.child_fd, maxsize)
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
    raise exc  # pylint: disable=raising-bad-type
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 1023: unexpected end of data

Error: There was a query error: Exception caught
Caught Exception, terminating workers
TRACE: 'utf8' codec can't decode byte 0xe2 in position 1023: unexpected end of data
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 67, in _call
    ret = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 2294, in create_multiprocessing
    local_master=parallel_data['local_master']
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1253, in create
    output = self.clouds[func](vm_)
  File "/usr/lib/python2.7/site-packages/salt/cloud/clouds/vmware.py", line 2936, in create
    out = __utils__['cloud.bootstrap'](vm_, __opts__)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 644, in bootstrap
    deployed = deploy_script(**deploy_kwargs)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 1762, in deploy_script
    if root_cmd(deploy_command, tty, sudo, **ssh_kwargs) != 0:
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2367, in root_cmd
    retcode = _exec_ssh_cmd(cmd, allow_failure=allow_failure, **kwargs)
  File "/usr/lib/python2.7/site-packages/salt/utils/cloud.py", line 2016, in _exec_ssh_cmd
    stdout, stderr = proc.recv()
  File "/usr/lib/python2.7/site-packages/salt/utils/vt.py", line 311, in recv
    return self._recv(maxsize)
  File "/usr/lib/python2.7/site-packages/salt/utils/vt.py", line 680, in _recv
    os.read(self.child_fd, maxsize)
  File "/usr/lib/python2.7/site-packages/salt/utils/stringutils.py", line 160, in to_unicode
    raise exc  # pylint: disable=raising-bad-type
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe2 in position 1023: unexpected end of data

Setup

Below is a map file we used where we got the error:

hou_vcp_em:

  - houcntrgis14.example.com:
      num_cpus: 4
      memory: 16GB
      devices:
        disk:
          Hard disk 2:
            size: 100
            thin_provision: True

  - houcntrgis15.example.com:
      num_cpus: 4
      memory: 16GB
      devices:
        disk:
          Hard disk 2:
            size: 100
            thin_provision: True

kty_vcp_em:

  - ktycntrgis14.example.com:
      num_cpus: 4
      memory: 16GB
      devices:
        disk:
          Hard disk 2:
            size: 100
            thin_provision: True

  - ktycntrgis15.example.com:
      num_cpus: 4
      memory: 16GB
      devices:
        disk:
          Hard disk 2:
            size: 100
            thin_provision: True

Steps to Reproduce Issue

In order to reproduce the issue, a map file like the one above is required. Once the map file is complete, run salt-cloud -P -m <map> -ldebug with the name of the map file in the "map" section.

Versions Report

Salt Version:
           Salt: 3000.2

Dependency Versions:
           cffi: 1.13.2
       cherrypy: unknown
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.28.4
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 2.7.5 (default, Apr  2 2020, 13:16:51)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4

System Versions:
           dist: centos 7.8.2003 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.8.2.el7.x86_64
         system: Linux
        version: CentOS Linux 7.8.2003 Core

The version of pyvmomi we are running is python2-pyvmomi-6.7.3-1.xxx.el7.noarch.

DmitryKuzmenko commented 4 years ago

Wow! That's interesting. The data being read have been cut by maxsize limit that produces a half of unicode character in the end of the buffer. https://github.com/saltstack/salt/blob/master/salt/utils/vt.py#L676 Thank you for catch a good bug @javaria105 !

sagetherage commented 4 years ago

@saltstack/team-cloud anyone have time to look into this issue in the next week or so?

saltstack / salt