[salt-cloud] Error actioning machines: 'ascii' codec can't encode character u'\xa0' in position 20

[webtrekker]

When trying to create a VMWare snapshot using --- salt-cloud -a create_snapshot syslog01 snapshot_name="mySaltSnap" description="Snap created via Saltcloud"

I get the error -- UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

… in my massive Google’ing this I came across this link http://stackoverflow.com/questions/28544686/unicodeencodeerror-ascii-codec-cant-encode-characters-in-position-0-5-ordin

So, from another suggestion that it might be pyVmomi I found this – /usr/lib/python2.7/site-packages/pyVmomi/__init__.py

And added these two lines after line 20 …

line 20: reload(sys) Added: sys.set Added: defaultencoding('utf-8')

sed -n '20,23p' /usr/lib/python2.7/site-packages/pyVmomi/__init__.py import sys reload(sys) sys.setdefaultencoding('utf-8')

Guess what … IT WORKED !!

On my SLES12 Salt-master: salt --version salt 2015.8.3 (Beryllium) salt-cloud --version salt-cloud 2015.8.3 (Beryllium) Note: my SLES12 Salt-master failed when trying to run salt --versions-report

Adding these lines also worked on my RHEL6.5 Salt-master sed -n '20,23p' /usr/lib/python2.6/site-packages/pyVmomi/__init__.py import sys reload(sys) sys.setdefaultencoding('utf-8')

salt --versions-report Salt Version: Salt: 2015.8.1

Dependency Versions: Jinja2: 2.2.1 M2Crypto: 0.20.2 Mako: Not Installed PyYAML: 3.11 PyZMQ: 14.5.0 Python: 2.6.6 (r266:84292, May 22 2015, 08:34:51) RAET: Not Installed Tornado: 4.2.1 ZMQ: 4.0.5 cffi: Not Installed cherrypy: 3.2.2 dateutil: 1.4.1 gitdb: Not Installed gitpython: Not Installed ioflo: Not Installed libnacl: Not Installed msgpack-pure: Not Installed msgpack-python: 0.4.6 mysql-python: Not Installed pycparser: Not Installed pycrypto: 2.6.1 pygit2: 0.20.3 python-gnupg: Not Installed smmap: Not Installed timelib: Not Installed

System Versions: dist: redhat 6.6 Santiago machine: x86_64 release: 2.6.32-504.8.1.el6.x86_64 system: Red Hat Enterprise Linux Server 6.6 Santiago

salt-cloud --version salt-cloud 2015.8.1 (Beryllium)

However, the above link suggests that what I added is a "nasty, nasty hack for people who don't understand encoding", which would be true of me, but maybe someone from the pyVmomi team or salt-cloud team can find a better work fix.

[jfindlay]

@webtrekker, thanks for the report. Are you working with any strings with non-ASCII chars? Is it possible for you to post your vmware cloud configs with the private info redacted?

@s0undt3ch, could this be fixed by salt-cloud handling the system encoding in the same way salt does if it does not do this already?

[webtrekker]

Here ya go ... cat /etc/salt/cloud.providers.d/vmware.conf vmware-config: driver: vmware user: 'VMWare.local\cfgmgt' password: 'reals3cure-trustme' url: '1.2.6.201'

[s0undt3ch]

@jfindlay if salt-cloud is not using the same approach that salt is, it should... and yeah @webtrekker, we should not rely on that hack... We need to find the deeper issue and handle it appropriately.

[s0undt3ch]

@webtrekker can we get the debug logs of that salt-cloud call? pass -l debug to the salt-cloud command. Of course, remove any sensitive data in those logs before posting...

[webtrekker]

Okay, I've taken out the hack and removed any sensitive data, hope this is workable ... As a side note, I've inspected all of the config files it reads/includes to validate that there weren't any special/hidden characters using cat -vT filename and the :set list in vi.

root@rlx9s # salt-cloud -l debug -a create_snapshot slx9e snapshot_name="mySaltSnap" description="Snap created via Saltcloud"
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: rlx9s.mycompany.com
[DEBUG   ] Missing configuration file: /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/vmware.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/vmware.conf
[DEBUG   ] Missing configuration file: /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/cmatrixbox.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/cmatrixbox.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] Could not LazyLoad rackspace.reboot
[DEBUG   ] Could not LazyLoad openstack.list_locations
[DEBUG   ] Could not LazyLoad rackspace.list_locations
The following virtual machines are set to be actioned with "create_snapshot":
  slx9e

Proceed? [N/y] y
... proceeding
[DEBUG   ] Could not LazyLoad vmware.optimize_providers
[DEBUG   ] The 'vmware' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] Could not LazyLoad rackspace.reboot
[DEBUG   ] Could not LazyLoad openstack.list_locations
[DEBUG   ] Could not LazyLoad rackspace.list_locations
[DEBUG   ] **PW***reals3cure-trustme
[DEBUG   ] **UN***VMWare.local\cfgmgt
[DEBUG   ] **URL***1.2.6.201
/usr/lib/python2.6/site-packages/urllib3/connectionpool.py:769: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
  InsecureRequestWarning)
[DEBUG   ] vm:sapsf1ap01-lnx82 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:Z-VRA-ubpe04.mycompany.com in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hbswxr183 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hsb9ci01-lnx56 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:SLES-grubtest in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:rlx9y-lnxrepo01 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:caprpt04-slx09 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:nasa13x in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:lnx46-splunk02 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapqf1ci02-rlxq2 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:devdcim-slx0h in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:nacq01-n01 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:nasa03x in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hpb3ap01-lnx54 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:emrdb01-slx0a in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sappr2a-lnx36-DCOM01072016 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hqb9ci01-rlx0s in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hbswxr510x in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:rlx9t in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:dbsliq1-slxs1 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:dbpf1-n02-slx08 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sappr2ap09-slxp9 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:lnex08-lnx11 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:lvmt01 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sappr3ap03-lnx71 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hbswxr506x in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hpb3ap04-rlxp4 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hqb3ap03-rlx16 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:lnxdcn01 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sappr3ap06-lnx79 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sappr3ap11-rlxp7 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapqo1-lnx10 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapqr4ap01-rlx06 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapql1ap01-lnx9b in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:Z-VRA-ubpe02.mycompany.com in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapqh1ap01-slx11 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:rlxgold-RHEL6.6-test in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:slx9s-htb401 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:nasa05_Doubletake in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hab3ap01-lnx48 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:hbswxr503x in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:sapdd2-lnx05 in provider:vmware is not in name list:set(['slx9e'])
[DEBUG   ] vm:saltydog in provider:vmware is not in name list:set(['slx9e'])
[ERROR   ] There was an error actioning machines: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/cloud/cli.py", line 244, in run
    ret = mapper.do_action(names, kwargs)
  File "/usr/lib/python2.6/site-packages/salt/cloud/__init__.py", line 1444, in do_action
    vm_name, driver, names
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

[s0undt3ch]

There we go... One of those 3 variables needs to be sdecoded, but that info is coming over the wire I believe, so, we'll have to know the content encoding...

[jfindlay]

@nmadhok has done the majority of the work on the salt-cloud vmware integration, so he may have some good information on that.

[nmadhok]

@jfindlay This doesn't seem to be a problem with the salt-cloud VMware driver. This seems to be a problem with the way salt-cloud handles Unicode support. I don't see the Unicode Error when I create a snapshot. @webtrekker may have to set the encoding type of his terminal to 'utf-8'. To fix this within salt-cloud, everything being read should be encoded to 'utf-8' instead of treating them as plain strings.

[nmadhok]

@webtrekker

Can you put the user in double quotes instead of single quotes in your VMware provider config file and escape the \?

vmware-config:
  driver: vmware
  user: "VMWare.local\\cfgmgt"
  password: 'reals3cure-trustme'
  url: '1.2.6.201'

It seems like your user may be causing the issue since it has \c in it.

[jfindlay]

@nmadhok, that is true, although salt does try to guess the system encoding, https://github.com/saltstack/salt/blob/v2015.8.3/salt/__init__.py#L33 and then use it to transcode strings coming from the system.

[nmadhok]

@jfindlay If encoding not found, then it sets it to ascii https://github.com/saltstack/salt/blob/v2015.8.3/salt/__init__.py#L57

@webtrekker Can you post the output of following command:

nitinmadhok@nitinmadhok:~$ echo $LANG
en_US.UTF-8

Also, can you open a python interpreter and print the output of following:

nitinmadhok@nitinmadhok:~$ python
Python 2.7.11 (default, Jan  4 2016, 10:43:13) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'

OR

nitinmadhok@nitinmadhok:~$ python -c 'import sys; print(sys.getdefaultencoding())'
ascii

Are you able to perform the following commands successfully?

nitinmadhok@nitinmadhok:~$ python -c 'print (u"voil\u00e0")'
voilà

nitinmadhok@nitinmadhok:~$ python -c 'print(u"\u2122")'
™

[webtrekker]

@nmadhok, it didn't like the double quotes --

[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/vmware.conf
[ERROR   ] Error parsing configuration file: /etc/salt/cloud.providers.d/vmware.conf - while scanning a double-quoted scalar
  in "<string>", line 3, column 9:
      user: "VMWare.local\cfgmgt"
            ^
found unknown escape character 'c'
  in "<string>", line 3, column 29:
      user: "VMWare.local\cfgmgt"

And on a by-guess-by-golly figured I'd try escaping the \c with \\c No joy

[DEBUG   ] (vim.fault.InvalidLogin) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   msg = 'Cannot complete login due to an incorrect user name or password.',
   faultCause = <unset>,
   faultMessage = (vmodl.LocalizableMessage) []
}
Cannot complete login due to an incorrect user name or password.

Also tried the username@ method ... also, no joy ... user: 'cfgmgt@VMWare.local'

[ERROR   ] There was an error actioning machines: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 246, in run
    ret = mapper.do_action(names, kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1444, in do_action
    vm_name, driver, names
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

Just to validate that the \cfgmgt might be the problem I can create a new user... any suggestions on what characters I shouldn't use?
Or is there another way to pass the username? I tried w/o the VMWare.local\, but that didn't work either as VMWare needs to know if the user is local or part of the AD domain.

[webtrekker]

I've also tried using the AD domain account -- user: 'NET\svc-EASalt', but had the same encoding error

[ERROR   ] There was an error actioning machines: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/cloud/cli.py", line 246, in run
    ret = mapper.do_action(names, kwargs)
  File "/usr/lib/python2.7/site-packages/salt/cloud/__init__.py", line 1444, in do_action
    vm_name, driver, names
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128)

so, seems \s won't work either if this is the root cause.

[lorengordon]

It looks to me like it's throwing the error while trying to print one of the vm_names. The error is being thrown by the log.debug function on vm_name. Got any VMs with slightly odd names?

https://github.com/saltstack/salt/blob/v2015.8.3/salt/cloud/__init__.py#L1443-L1444

[webtrekker]

@lorengordon, nothing out of the ordinary ... no special characters other than these three - , _, or a . , most of our names are in this format xyz123-abc456 or Z-xyz123_dt

[nmadhok]

@lorengordon The vm names should be encoded and printed as unicode strings?

@webtrekker Can you post the output of following command:

nitinmadhok@nitinmadhok:~$ echo $LANG
en_US.UTF-8

Also, can you open a python interpreter and print the output of following:

nitinmadhok@nitinmadhok:~$ python
Python 2.7.11 (default, Jan  4 2016, 10:43:13) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'

OR

nitinmadhok@nitinmadhok:~$ python -c 'import sys; print(sys.getdefaultencoding())'
ascii

Are you able to perform the following commands successfully?

nitinmadhok@nitinmadhok:~$ python -c 'print (u"voil\u00e0")'
voilà

nitinmadhok@nitinmadhok:~$ python -c 'print(u"\u2122")'
™

[lorengordon]

@nmadhok: I don't know about should, but it has to be throwing up on vm_name because the other two values are printed just fine in prior iterations of the loop (or so it appears from the log output in this thread). So, worth a try? We'd have to look up the allowed VM name characters, I guess, to be sure. String encodings give me a headache.

[nmadhok]

@webtrekker Can you change line 1444/1445, in function do_action in /usr/lib/python2.6/site-packages/salt/cloud/__init__.py to the following:

                            log.debug(
                                'vm:{0} in provider:{1} is not in name '
                                'list:\'{2}\''.format(vm_name.encode('utf-8'), driver, names)
                            )

Try this and let me know if you still see the error?

[webtrekker]

@nmadhok,

1442                     if vm_name not in names:
1443             #            log.debug('vm:{0} in provider:{1} is not in name list:{2!r}'.format(
1444             #                vm_name, driver, names
1445             #            ))
1446                         log.debug(
1447                                 'vm:{0} in provider:{1} is not in name '
1448                                 'list:\'{2}\''.format(vm_name.encode('utf-8'), driver, names)
1449                             )
1450                         continue

And Ladies and Gentlemen, members of the press, and Distinguished Guests .... We have a Winner!

salt-cloud -l debug -a create_snapshot slx9e snapshot_name="mySaltSnap" description="Snap created via Saltcloud"
[DEBUG   ] Reading configuration from /etc/salt/cloud
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/api.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/api.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/fileserver.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/fileserver.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/reactor.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/reactor.conf
[DEBUG   ] Including configuration from '/etc/salt/master.d/susemanager.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/susemanager.conf
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers
[DEBUG   ] Including configuration from '/etc/salt/cloud.providers.d/vmware.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.providers.d/vmware.conf
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles
[DEBUG   ] Including configuration from '/etc/salt/cloud.profiles.d/cmatrixbox.conf'
[DEBUG   ] Reading configuration from /etc/salt/cloud.profiles.d/cmatrixbox.conf
[DEBUG   ] Configuration file path: /etc/salt/cloud
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] salt-cloud starting
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
The following virtual machines are set to be actioned with "create_snapshot":
  slx9e

Proceed? [N/y] y
... proceeding
[DEBUG   ] Could not LazyLoad vmware.optimize_providers
[DEBUG   ] The 'vmware' cloud driver is unable to be optimized.
[DEBUG   ] Could not LazyLoad parallels.avail_sizes
[DEBUG   ] LazyLoaded parallels.avail_locations
[DEBUG   ] LazyLoaded proxmox.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.destroy
[DEBUG   ] Could not LazyLoad saltify.avail_sizes
[DEBUG   ] Could not LazyLoad saltify.avail_images
[DEBUG   ] Could not LazyLoad saltify.avail_locations
[DEBUG   ] LazyLoaded rackspace.reboot
[DEBUG   ] LazyLoaded openstack.list_locations
[DEBUG   ] LazyLoaded rackspace.list_locations
[DEBUG   ] vm:sf1ap01-lnx82 in provider:vmware is not in name list:'set(['slx9e'])'

.... huge list ... continues...

[INFO    ] [ slx9e ] Waiting for create snapshot task to finish [0 s]
[INFO    ] [ slx9e ] Waiting for create snapshot task to finish [5 s]
[INFO    ] [ slx9e ] Waiting for create snapshot task to finish [10 s]
[INFO    ] [ slx9e ] Successfully completed create snapshot task in 11 seconds
[DEBUG   ] LazyLoaded nested.output
mycompany-vmware-config:
    ----------
    vmware:
        ----------
        slx9e:
            ----------
            Snapshot created successfully:
                ----------
                created:
                    2016-01-13 19:51:43
                description:
                    Snap created via Saltcloud
                name:
                    mySaltSnap
                path:
                    /mySaltSnap
                state:
                    poweredOn

[nmadhok]

@webtrekker This is a temporary hack and there should be a better way to handle this. @s0undt3ch @jfindlay Salt doesn't seem to be setting the encoding to utf-8 correctly.

[nmadhok]

@webtrekker Can you post the output of following command:

nitinmadhok@nitinmadhok:~$ echo $LANG
en_US.UTF-8

Also, can you open a python interpreter and print the output of following:

nitinmadhok@nitinmadhok:~$ python
Python 2.7.11 (default, Jan  4 2016, 10:43:13) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'

OR

nitinmadhok@nitinmadhok:~$ python -c 'import sys; print(sys.getdefaultencoding())'
ascii

Are you able to perform the following commands successfully?

nitinmadhok@nitinmadhok:~$ python -c 'print (u"voil\u00e0")'
voilà

nitinmadhok@nitinmadhok:~$ python -c 'print(u"\u2122")'
™

[webtrekker]

@nmadhok, could this be the underlying issue ...

 echo $LANG
POSIX

Here is the rest of what you requested, seems all work as expected ...

 python
Python 2.7.9 (default, Dec 21 2014, 11:02:59) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>>

and

python -c 'print (u"voil\u00e0")'
voilà

and

python -c 'print(u"\u2122")'
™

[nmadhok]

@webtrekker Can you do the following and try again:

export LANG=en_US.UTF-8

[nmadhok]

@jfindlay This is not an upstream bug.

[webtrekker]

Okay... exported LANG and reset the __init__.py back to original. That combo still fails. The code changes works with either POSIX or en_US.UTF-8

Well... hold that ... something is going sideways ... the snapshot is now grabbing a completely diff machine than what was specified on the command line ..

The following virtual machines are set to be actioned with "create_snapshot":
  slx9e

Proceed? [N/y] y
... proceeding

but...

[INFO    ] [ sf1ap01-lnx82 ] Waiting for create snapshot task to finish [0 s]
^C
Exiting gracefully on Ctrl-c

[nmadhok]

@webtrekker Alright. Does it get fixed when you export LANG back to POSIX?

[webtrekker]

Okay... I must have foobar'd the code when setting it back to original. My apologies. I put the changes back in and reset the system to use LANG=POSIX. Back to working as before with the edited code.

Then, redid the LANG=en_US.UTF-8 plus edited code test .. this combo also works. Then, redid the LANG=en_US.UTF-8 plus original code test ... this combo does fail.

[nmadhok]

@rallytime @s0undt3ch @jfindlay What do you suggest we should do here? We have a fix but I don't like it honestly. In my opinion, salt should handle the encoding rather than encoding strings everytime in debug statements.

[s0undt3ch]

What http lib dies the VMware library use? Our HTTP lib? A VMware specific lib? Ie, what do we use to query the VMware api.

[nmadhok]

It uses requests. We use pyVmomi to query using VMware API

[s0undt3ch]

OK, then any data coming from those needs to be turned into Unicode. Requests will tell you the encoding used, don't know about the other...

... Or, was this a configuration issue? Sorry I've been following up on my phone...

[s0undt3ch]

If that log call is something like:

log.debug(six.u('....blah...'))

Does it work?

[webtrekker]

@s0undt3ch, if that is something you'd like me to try just let me know what I need to change and where. Note: ... I'm and admin, not a coder :smile:

So, it kinda sounds like we have 2 solutions

• edit /usr/lib/python2.6/site-packages/pyVmomi/__init__.py add 2 lines at line 21.

reload(sys)
sys.setdefaultencoding('utf-8')

or

• edit /usr/lib/python2.6/site-packages/salt/cloud/__init__.py and change the log.debug stanza.

From what I've gathered so far is that neither are ideal.
Is that a fair recap?

[s0undt3ch]

For now, try this debug call and test please:

1442                     if vm_name not in names:
1443             #            log.debug('vm:{0} in provider:{1} is not in name list:{2!r}'.format(
1444             #                vm_name, driver, names
1445             #            ))
1446                         log.debug(
1447                             u'vm:{0} in provider:{1} is not in name list:\'{2}\''.format(vm_name, driver, names)
1448                         )
1449                         continue

[s0undt3ch]

Oh, and option 1, reload(sys) that's a no, no :smile:

[webtrekker]

@s0undt3ch, that debug call works as well.

[s0undt3ch]

So, what we've learned here is that the underlying libraries return Unicode, as they should, and, the problem is that the format call of the debug message does not assume that any of those arguments could be Unicode. The immediate fix is wrapping the log message string in six.u(), before formatting... log.debug(six.u('log msg').format(args)).

The long run fix is stopping to use format, just on log messages by using the %s place holders, pass the arguments as arguments to the log call, and handle Unicode issues and conversion on our log handler....

[webtrekker]

so.... should I just leave the last change in place until the next revision? BTW, I'm fine w/that just wanted to make sure no one was waiting on me for something. Thanks everyone for your help... this was my first time being involved in any project code.

[rallytime]

@webtrekker Yeah that should be fine. I'll be sure to take care of this as @s0undt3ch suggests above for the next version. Thanks for your testing! It is much appreciated.

[rallytime]

This should be fixed with #30374.

[rallytime]

In my first PR, my brain translated "pyVmomi" to "VMware Cloud Driver" and I changed the formatted logging strings to unicode as @s0undt3ch described above. Right after I submitted it, I realized that the change you made locally that is working, and the where the stacktrace was pointing, was supposed to be made in cloud/__init__.py. The second (#30375) PR in the list fixes the init file.