search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.15k stars 5.47k forks source link

Fail to use 'highstate' outputter explicitly #29796

Closed vutny closed 8 years ago

vutny commented 8 years ago

Hi,

I've configured the master to use custom output module and it works great. When I tired to get back to 'highstate' outputter once for a while, it failed miserably:

# salt --out=highstate '*' state.highstate
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 137, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 421, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 137, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 421, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 137, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 421, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 129, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'list' object has no attribute 'get'

By the way, I've got the same bunch of tracebacks when I tried to set the output: highstate option in the master's config file.

# salt --versions-report 
Salt Version:
           Salt: 2015.8.1

Dependency Versions:
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: 0.8.6
       cherrypy: Not Installed
       dateutil: 2.4.2
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: 0.21.4
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 3.10.0-229.7.2.el7.x86_64
         system: CentOS Linux 7.1.1503 Core
jfindlay commented 8 years ago

@vutny, thanks for reporting. Do you have any more information on what may cause the failure condition to happen to help narrow in on the cause of the issue?

vutny commented 8 years ago

Yep, I did a deeper dive into it.

Basic module test works:

# salt --out=highstate salt test.ping
salt:
    True

Next, I've created minimal test.sls:

always-passes:
  test.succeed_without_changes:
    - name: foo

The command salt --out=highstate salt state.sls test works as expected.

This sls produce valid output as well:

always-fails:
  test.fail_without_changes:
    - name: foo

Let's try another approach, run an invalid command:

# salt --out=highstate WRONG_MINION_ID state.sls test
No minions matched the target. No command was sent, no jid was assigned.
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'NoneType' object has no attribute 'rstrip'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
AttributeError: 'NoneType' object has no attribute 'rstrip'

Hey, this is something new! :-) But works without setting the outputter:

# salt WRONG_MINION_ID state.sls test
No minions matched the target. No command was sent, no jid was assigned.
ERROR: No return received

Go back to the test.sls and put the state that would return some changes:

always-changes-and-succeeds:
  test.succeed_with_changes:
    - name: foo

Apply the state:

# salt --out=highstate salt state.sls test
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 139, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 423, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'
[ERROR   ] Nested output failed: 
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'
[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 139, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 423, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 454, in salt_main
    client.run()
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 204, in run
    self._output_ret(ret_, out)
  File "/usr/lib/python2.7/site-packages/salt/cli/salt.py", line 303, in _output_ret
    salt.output.display_output(ret, out, self.config)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 86, in display_output
    display_data = try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 139, in _format_host
    schanged, ctext = _format_changes(ret['changes'])
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 423, in _format_changes
    __opts__)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 164, in out_format
    return try_printout(data, out, opts)
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 46, in try_printout
    return get_printout('raw', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'

This is it. Got exactly the same result with sls:

always-changes-and-fails:
  test.fail_with_changes:
    - name: foo
vutny commented 8 years ago

Almost forgot, I've updated Salt to the 2015.8.3 version.

jfindlay commented 8 years ago

@vutny, thanks for the excellent info.

roock commented 8 years ago

I'm seeing the same behavior. If nothing is changed or only failed state are returned by minion, everything works as expected. If there have been changes, it crashes.

Tested with:

always-changes-and-succeeds:
  test.succeed_with_changes:
    - name: foo

The error:

$ time sudo salt --out='highstate' --state-output='changes' '*' state.highstate
[ERROR   ] Nested output failed:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/salt/output/__init__.py", line 43, in try_printout
    return get_printout('nested', opts)(data).rstrip()
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 85, in output
    return _format_host(host, hostdata)[0]
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in _format_host
    key=lambda k: data[k].get('__run_num__', 0)):
  File "/usr/lib/python2.7/site-packages/salt/output/highstate.py", line 131, in <lambda>
    key=lambda k: data[k].get('__run_num__', 0)):
AttributeError: 'str' object has no attribute 'get'

Version Info:

Salt Version:
           Salt: 2015.8.3

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: 0.21.1
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.1.0
         Python: 2.7.11 (default, Dec 11 2015, 05:34:18)
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.3
           cffi: 1.4.2
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist:
        machine: armv7l
        release: 4.2.5-258
rallytime commented 8 years ago

Hi everyone! I finally was able to track down the culprit for this issue. A fix is included in #33215.

@vutny please give that a try when you get a moment and let me know how it goes.