search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Install Salt from the Salt package repositories here:
https://docs.saltproject.io/salt/install-guide/en/latest/
Apache License 2.0
14.18k stars 5.48k forks source link

salt '*' state.highstate UnicodeDecodeError #37906

Closed hujixiang2016 closed 6 years ago

hujixiang2016 commented 7 years ago

Description of Issue/Question

I perform salt '*' state.highstate ,there are a ERROR ,I thought it was a coding problem, modify the sitecustomize.py file, but without success,plese help me

Setup

(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).) [root@localhost lib]# more /usr/lib/python2.6/site-packages/yum/sitecustomize.py

encoding=utf8

import sys reload(sys) sys.setdefaultencoding('utf8')

Steps to Reproduce Issue

(Include debug logs if possible and relevant.) [root@localhost lib]# salt '*' state.highstate [ERROR ] An un-handled exception was caught by salt's global exception handler: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 77: ordinal not in range(128) Traceback (most recent call last): File "/usr/bin/salt", line 10, in salt_main() File "/usr/lib/python2.6/site-packages/salt/scripts.py", line 349, in salt_main client.run() File "/usr/lib/python2.6/site-packages/salt/cli/salt.py", line 210, in run self._outputret(ret, out) File "/usr/lib/python2.6/site-packages/salt/cli/salt.py", line 300, in _output_ret salt.output.display_output(ret, out, self.config) File "/usr/lib/python2.6/site-packages/salt/output/init.py", line 94, in display_output display_data = try_printout(data, out, opts) File "/usr/lib/python2.6/site-packages/salt/output/init.py", line 46, in try_printout return get_printout(out, opts)(data).rstrip() File "/usr/lib/python2.6/site-packages/salt/output/highstate.py", line 84, in output return _format_host(host, hostdata)[0] File "/usr/lib/python2.6/site-packages/salt/output/highstate.py", line 112, in _format_host .format(hcolor, err, colors))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 77: ordinal not in range(128) Traceback (most recent call last): File "/usr/bin/salt", line 10, in salt_main() File "/usr/lib/python2.6/site-packages/salt/scripts.py", line 349, in salt_main client.run() File "/usr/lib/python2.6/site-packages/salt/cli/salt.py", line 210, in run self._outputret(ret, out) File "/usr/lib/python2.6/site-packages/salt/cli/salt.py", line 300, in _output_ret salt.output.display_output(ret, out, self.config) File "/usr/lib/python2.6/site-packages/salt/output/init.py", line 94, in display_output display_data = try_printout(data, out, opts) File "/usr/lib/python2.6/site-packages/salt/output/init.py", line 46, in try_printout return get_printout(out, opts)(data).rstrip() File "/usr/lib/python2.6/site-packages/salt/output/highstate.py", line 84, in output return _format_host(host, hostdata)[0] File "/usr/lib/python2.6/site-packages/salt/output/highstate.py", line 112, in _format_host .format(hcolor, err, colors))) UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 77: ordinal not in range(128)

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) Salt: 2015.5.10 Python: 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) Jinja2: unknown M2Crypto: 0.20.2 msgpack-python: 0.4.6 msgpack-pure: Not Installed pycrypto: 2.0.1 libnacl: Not Installed PyYAML: 3.10 ioflo: Not Installed PyZMQ: 14.3.1 RAET: Not Installed ZMQ: 3.2.5 Mako: 0.3.4 Tornado: Not Installed timelib: Not Installed dateutil: Not Installed

gtmanfred commented 7 years ago

Can you try enabling yaml_utf8: True in your master config? https://docs.saltstack.com/en/latest/ref/configuration/master.html#yaml-utf8

If you could also figure out which state is causing this and provide us with that state so we can replicate this issue, that would be awesome!

Thanks, Daniel

qurczak commented 7 years ago

Another methot how to reproduce this bug:

yaml_utf8 enabled in master

Error:

$ salt asus-p7f-x state.sls pacman test=True
asus-p7f-x:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/salt/minion.py", line 1412, in _thread_return
        return_data = executor.execute()
      File "/usr/lib/python2.7/site-packages/salt/executors/direct_call.py", line 28, in execute
        return self.func(*self.args, **self.kwargs)
      File "/usr/lib/python2.7/site-packages/salt/modules/state.py", line 1039, in sls
        ret = st_.state.call_high(high_, orchestration_jid)
      File "/usr/lib/python2.7/site-packages/salt/state.py", line 2268, in call_high
        chunks = self.compile_high_data(high, orchestration_jid)
      File "/usr/lib/python2.7/site-packages/salt/state.py", line 1323, in compile_high_data
        chunks = self.order_chunks(chunks)
      File "/usr/lib/python2.7/site-packages/salt/state.py", line 1260, in order_chunks
        chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
      File "/usr/lib/python2.7/site-packages/salt/state.py", line 1260, in <lambda>
        chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))
    UnicodeEncodeError: 'ascii' codec can't encode characters in position 2-4: ordinal not in range(128)
lut 21 12:41:02 salt-minion[1941]: [WARNING ] The minion function caused an exception
lut 21 12:41:02 salt-master[21848]: [ERROR   ] Received minion error from [asus-p7f-x]: The minion function caused an exception

File /srv/salt/pacman/init.sls

$ cat init.sls 
example1:
  cmd.run:
    - name: |
        # ąść <--- some national chars
        id

Version:

Salt Version:
           Salt: 2016.11.2

Dependency Versions:
           cffi: 1.9.1
       cherrypy: unknown
       dateutil: 2.6.0
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: 1.6.7
         Jinja2: 2.9.5
        libgit2: 0.24.3
        libnacl: 1.5.0
       M2Crypto: 0.24.0
           Mako: 1.0.6
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: 1.2.5
      pycparser: 2.17
       pycrypto: 2.6.1
         pygit2: 0.24.2
         Python: 2.7.13 (default, Dec 21 2016, 07:16:46)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: 0.6.6
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.2
            ZMQ: 4.2.0

System Versions:
           dist:   
        machine: x86_64
        release: 4.9.9-1-ARCH
         system: Linux
        version: Not Installed
lubyou commented 7 years ago

Hi,

I can confirm the second method.

This is happening because there are a few calls to format() in state.py, and those calls throw exceptions when you mix unicode and byte strings.

This is one of the calls: https://github.com/saltstack/salt/blob/develop/salt/state.py#L556

If the format template strings are marked as unicode, it works as expected, ie chunks.sort(key=lambda chunk: (chunk['order'], u'{0[state]}{0[name]}{0[fun]}'.format(chunk))) instead of chunks.sort(key=lambda chunk: (chunk['order'], '{0[state]}{0[name]}{0[fun]}'.format(chunk)))

absmith82 commented 6 years ago

I am also having this problem with the salt.gitfs.pygit2 formula as of 2018.3.0 this was not an issue in 2017.7.5

salt-call state.apply salt.gitfs.pygit2
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1878, in call
    **cdata['kwargs'])
  File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1823, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/salt/states/archive.py", line 1013, in extracted
    verbose=True)
  File "/usr/lib/python2.7/dist-packages/salt/modules/archive.py", line 419, in list_
    if x.count('/') == 0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 38: ordinal not in range(128)

Pillar info

salt:
  gitfs:
    pygit2:
      install_from_source: True
      version: 0.26.0
      git:
        require_state: False
        install_from_package: git
      libgit2:
        version: 0.26.0
        install_from_source: True
        build_parent_dir: /usr/src/
        download_hash: 6ea75a8a5745a7b2a14d3ed94486e761

note that pygit2 is already installed from 2017.7 release

Salt Version:
           Salt: 2018.3.0

Dependency Versions:
           cffi: 1.11.0
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: 0.26.0
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.18
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.26.0
         Python: 2.7.12 (default, Dec  4 2017, 14:50:18)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-119-generic
         system: Linux
        version: Ubuntu 16.04 xenial
dawidmalina commented 6 years ago

I have exact issue

          ID: extract-package
    Function: archive.extracted
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1905, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1830, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/archive.py", line 1003, in extracted
                  verbose=True)
                File "/usr/lib/python2.7/dist-packages/salt/modules/archive.py", line 419, in list_
                  if x.count('/') == 0]
              UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 38: ordinal not in range(128)
     Started: 22:06:49.898613
    Duration: 25030.856 ms
     Changes:

And my state looks like this:

extract-package:
  archive.extracted:
    - name: /tmp
    - source: salt://{{ slspath }}/files/v0.26.0.tar.gz
    - if_missing: /tmp/libgit2-0.26.0
    - archive_format: tar

This was working perfectly with 2017.7.x but now we can't move on to 2018.3.x :(

absmith82 commented 6 years ago

This seems to be fixed in 2018.3.1 and 2018.3.2

rallytime commented 6 years ago

Thanks for updating @absmith82! We did a lot of unicode work for the 2018.3.0 release, and more individual/follow up fixes for 2018.3.1 and 2018.3.2.

I'm going to close this. If people have more specific issues, please feel free to file a new issue so we can get more eyes on it. Thank you everyone!

remyd1 commented 6 years ago

Hi,

I have the same kind of error:

          ID: dell_omsa_repo
    Function: pkgrepo.managed
        Name: deb http://linux.dell.com/repo/community/ubuntu xenial openmanage
      Result: False
     Comment: An exception occurred in this state: Traceback (most recent call last):
                File "/usr/lib/python2.7/dist-packages/salt/state.py", line 1931, in call
                  **cdata['kwargs'])
                File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1943, in wrapper
                  return f(*args, **kwargs)
                File "/usr/lib/python2.7/dist-packages/salt/states/pkgrepo.py", line 398, in managed
                  ppa_auth=kwargs.get('ppa_auth', None)
                File "/usr/lib/python2.7/dist-packages/salt/modules/aptpkg.py", line 1651, in get_repo
                  repos = list_repos()
                File "/usr/lib/python2.7/dist-packages/salt/modules/aptpkg.py", line 1603, in list_repos
                  repo['line'] = salt.utils.pkg.deb.strip_uri(source.line.strip())
                File "/usr/lib/python2.7/dist-packages/salt/utils/pkg/deb.py", line 38, in strip_uri
                  for x in ('http://', 'https://', 'ftp://')):
                File "/usr/lib/python2.7/dist-packages/salt/utils/pkg/deb.py", line 38, in <genexpr>
                  for x in ('http://', 'https://', 'ftp://')):
              UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)
     Started: 14:04:46.827710
    Duration: 80.0000000745 ms
     Changes:

I had this error on salt 2018.3.2 and tried to switch to develop branch and I get the same error.

Salt Version:
           Salt: 2018.11.0-206-gb125901

Dependency Versions:
           cffi: 1.7.0
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 2.0.6
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.27.0
           Mako: 1.0.7
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: 1.3.10
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-33-generic
         system: Linux
        version: Ubuntu 18.04 bionic

My state is working on previous release, that is to say 2017.7.2 and 2017.7.3.

My state is the following:

{% if grains['os_family'] == 'Debian' %}
 {% set os = grains['os']|lower %}
 {% set release = grains['oscodename'] %}
 {% set majorrelease = grains['osmajorrelease'] %}
 # no recent package for ubuntu > 16
 {% if majorrelease > 16 %}
  {% set release = "xenial" %}
 {% endif %}

dell_omsa_repo:
  pkgrepo.managed:
    - humanname: linux.dell.com
    - name: deb http://linux.dell.com/repo/community/{{ os }} {{ release }} openmanage
    - file: /etc/apt/sources.list.d/linux.dell.com.sources.list
    - keyserver: pool.sks-keyservers.net
    - keyid: 1285491434D8786F

{% elif grains['os_family'] == 'RedHat' %}

dell_script_repo:
  cmd.run:
    - name: "curl -s http://linux.dell.com/repo/hardware/dsu/bootstrap.cgi | bash"
    - cwd: /
    - unless: ls /opt/curldell

touch /opt/curldell:
  cmd.run:
    - creates: /opt/curldell

{% endif %}

Any help would be appreciated.

Kind regards, Rémy