search

saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.17k stars 5.48k forks source link

[2019.2.0] Unhandled AttributeError: salt.defaults.exitcodes has no attribute EX_STATE_FAILURE #52640

Open vosscodes opened 5 years ago

vosscodes commented 5 years ago

Description of Issue/Question

I've run into a git/cmd failure followed by unhandled global exception on Suse Tumbleweed. Salt was updated to 2019.2 in an earlier state, and this did not repeat on a second run, so I believe this is related to cmdmod.py and updating Salt, as suggested in #52017

Setup

Masterless minion calling itself locally with salt-call

# /etc/salt/minion
file_client: local
pillar_merge_lists: True

file_roots:
  base:
    - /srv/salt/states
pillar_roots:
  base:
    - /srv/salt/pillar
# jinja template, first in the list of 6-10 identical states
salt_vim:
  git.latest:
    - name: https://github.com/saltstack/salt-vim
    - target: /home/user/directory
    - user: user 
    - branch: master
    - require:
      - state_name

# expected culprits, first cmd states after the git.latest states
user_nvm:
  cmd.script:
    - source: https://raw.githubusercontent.com/creationix/nvm/v0.34.0/install.sh
    - runas: user
    - unless:
        - test -f /home/user/.nvm/nvm.sh

user_nodejs:
  cmd.run:
    - name: nvm install node --latest-npm
    - runas: user
    - onchanges:
      - user_nvm

Error

This first trace repeats for each git state. The command returns 1 again when repeating the job a second time, but there was no traceback and the states/job finished normally.

[ERROR   ] Command '['git', 'config', '--global', '--get-regexp', 'filter\\.lfs\\.']' failed with return code: 1
[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/git.py", line 774, in latest
    output_encoding=output_encoding)
  File "/usr/lib/python3.7/site-packages/salt/modules/git.py", line 3951, in remote_refs
    output_encoding=output_encoding)['stdout']
  File "/usr/lib/python3.7/site-packages/salt/modules/git.py", line 386, in _git_run
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

Several very similar errors related to cmd.py show up next:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 916, in run
    name, timeout=timeout, python_shell=True, **cmd_kwargs
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2069, in run_all
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

These two are identical, excepting line 896 vs 1165 of cmd.py:

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 896, in run
    cret = mod_run_check(cmd_kwargs, onlyif, unless, creates)
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 374, in mod_run_check
    cmd.append(__salt__['cmd.retcode'](entry, ignore_retcode=True, python_shell=True, **cmd_kwargs))
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 2265, in retcode
    **kwargs)
  File "/usr/lib/python3.7/site-packages/salt/modules/cmdmod.py", line 650, in _run
    proc = salt.utils.timed_subprocess.TimedProc(cmd, **new_kwargs)
  File "/usr/lib/python3.7/site-packages/salt/utils/timed_subprocess.py", line 45, in __init__
    if kwargs.get('shell', False):
  File "/usr/lib64/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.7/subprocess.py", line 1523, in _execute_child
    raise child_exception_type(err_msg)
subprocess.SubprocessError: Exception occurred in preexec_fn.

[ERROR   ] An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/salt/state.py", line 1911, in call
    # allow setting the OS environ also make use of the "env"
  File "/usr/lib/python3.7/site-packages/salt/loader.py", line 1832, in wrapper
    start = time.time()
  File "/usr/lib/python3.7/site-packages/salt/states/cmd.py", line 1165, in script
    run_check_cmd_kwargs, onlyif, unless, creates
    (snip)

Finally, an AttributeError occurs:

[ERROR   ] An un-handled exception was caught by salt's global exception handler:
AttributeError: module 'salt.defaults.exitcodes' has no attribute 'EX_STATE_FAILURE'
Traceback (most recent call last):
  File "/usr/bin/salt-call", line 11, in <module>
    salt_call()
  File "/usr/lib/python3.7/site-packages/salt/scripts.py", line 400, in salt_call
    '''
  File "/usr/lib/python3.7/site-packages/salt/cli/call.py", line 57, in run
    caller.run()
  File "/usr/lib/python3.7/site-packages/salt/cli/caller.py", line 134, in run
    profiling_enabled = self.opts.get('profiling_enabled', False)
  File "/usr/lib/python3.7/site-packages/salt/cli/caller.py", line 212, in call
    sys.stderr.write(
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 749, in apply_
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 1061, in highstate
    st_ = salt.state.HighState(opts,
  File "/usr/lib/python3.7/site-packages/salt/modules/state.py", line 108, in _set_retcode
    __context__['retcode'] = salt.defaults.exitcodes.EX_STATE_FAILURE
AttributeError: module 'salt.defaults.exitcodes' has no attribute 'EX_STATE_FAILURE'

Versions Report

$ salt-call --versions-report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 3.7.2
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.2 (default, Dec 30 2018, 16:18:15) [GCC]
   python-gnupg: 0.4.4
         PyYAML: 5.1
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 5.0.7-1-default
         system: Linux
        version: Not Installed
waynew commented 5 years ago

Hey, thanks for the report! This is super interesting - There's a specific commit where this EX_STATE_FAILURE was introduced - I'm pretty sure the theory is correct that upgrading salt during a state run is what causes the problem, mainly due to a race condition when we're doing lazy importing.

I have some theories about how to reproduce this, but at least as far as I've found we don't currently have any guarantees about upgrading salt in the middle of running salt (or a highstate, anyway).

vosscodes commented 5 years ago

Hey, forgot to reply to this one. If you need any help testing on this, I'd be happy to lend a hand.

I don't recall seeing any guarantees either, but I think this will remain an issue as any pkg.uptodate state will trigger this if Salt has released an update without the user knowing. If it's not possible to guarantee upgrading during a highstate, would excluding Salt packages from pkg.uptodate states be possible? I wasn't able to get it working with kwargs.

General excludes (#27894) and/or a boolean value like git.latest's force_refresh to force (not) updating Salt packages would be extremely useful for enterprise use.

waynew commented 5 years ago

I'm a fan of that idea/approach. I'm not sure if, with our current architecture, we could make any guarantees for an update during a state run.

If there's a way to exclude Salt from a non-verbose update, that seems like a great idea to me.

arthurzenika commented 4 years ago

Bumping into this bug too, have you found a fix or a way round it ?

vosscodes commented 4 years ago

Update Salt, restart the services, and highstate again. This happens when salt updates itself during a highstate, which should succeed but will cause everything after to fail.

noelmcloughlin commented 4 years ago

cmd.run is throwing this exception on MacOS 10.15.5/Darwin 19.5.0 with Salt 3001. Is there any workaround?

[INFO    ] Executing command 'curl -Lo /usr/local/devspace-v4.13.1/bin/devspace https://github.com/devspace-cloud/devspace/releases/download/v4.13.1/devspace-darwin-amd64' in group 'staff' in directory '/Users/myusername'
[DEBUG   ] env command: ['sudo', '-g', 'staff', '-i', '--', '/usr/local/Cellar/salt/3001/libexec/bin/python3.8']
[ERROR   ] Exception occurred in preexec_fn.
markwhaite commented 4 years ago

@noelmcloughlin I am running 3001 on both master and mac minion. Mac OS X 10.15.5-10 and cmd.run seems to be working for me. me@salt:~$ sudo salt -G 'os:macos' cmd.run 'uptime' mac: 10:24 up 21 mins, 2 users, load averages: 11.49 12.17 11.49 However, I am running salt installed via the pkg installer and not brew. I am not sure if that could be an issue.