saltstack / salt

Software to automate the management and configuration of any infrastructure or application at scale. Get access to the Salt software package repository here:
https://repo.saltproject.io/
Apache License 2.0
14.1k stars 5.47k forks source link

salt-call state.highstate exiting with "Exiting gracefully on Ctrl-c" message #31280

Closed rambli closed 6 years ago

rambli commented 8 years ago

I'm facing the following issue running a highstate command with a very simple top file: Salt version: 2015.8.5 (Beryllium) Host: Windows 2012R2 VM Single minion running in masterless mode

C:\salt\srv\salt>salt-call state.highstate -l debug --local
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] Using cached minion ID from c:\salt\conf\minion_id: WIN-M9GCE6IBR1K.xyz.net
[DEBUG   ] Configuration file path: c:\salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged
.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.highstate
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded roots.envs

Exiting gracefully on Ctrl-c

Upon digging into salt code by following trace output, turns out saltutil.is_running check is the issue. I verified that upon commenting the state function running check below, my top.sls gets properly processed on a highstate call.

salt/modules/state.py: Line 119
active =  __salt__['saltutil.is_running']('state.*')

Even running this raises the "Exiting gracefully.." message:

$> salt-call saltutil.is_running state.highstate
local:
Exiting gracefully on Ctrl-c

I am digging in further to find out why attempting to get list of running jobs on this minion is raising a KeyboardInterrupt exception. Any ideas why checking state with saltutil.is_running is raising this error?

Although I'm fairly certain my state files have nothing to do with this issue, here they are for reference: top.sls:

base:
  '*':
    - service

service.sls:

Power:
  service.running:
    - enable: True
    - reload: True

Salt version info:

Salt Version:
           Salt: 2015.8.5

Dependency Versions:
         Jinja2: 2.8
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
         Python: 2.7.8 (default, Jun 30 2014, 16:08:48) [MSC v.1500 64 bit (AMD6
4)]
           RAET: Not Installed
        Tornado: 4.3
            ZMQ: 4.1.2
           cffi: 1.5.0
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
          ioflo: Not Installed
        libgit2: Not Installed
        libnacl: 1.4.4
   msgpack-pure: Not Installed   
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.9.0
        timelib: Not Installed

System Versions:
           dist:
        machine: AMD64
        release: 2012Server
         system: 2012Server 6.2.9200  Multiprocessor Free
jfindlay commented 8 years ago

@rambli, thanks for reporting. Is it possible for you to experiment with some different configurations, like, for example a simpler state:

test:
  test.succeed_without_changes:
    - name: test
rambli commented 8 years ago

@jfindlay Unfortunately that did not make any difference :( I don't think the state files themselves are a problem.

Issue arises when executing the running() function within salt/utils/minion.py.. the act of finding running jobs on a minion is somehow causing a SIGINT to be raised from somewhere (which I'm guessing is the reason for seeing the "Exiting gracefully on Ctrl-c" message)

Here is some debug with the --hard-crash flag set.

C:\salt\srv\salt>salt-call state.highstate --file-root=.  --local -l debug --hard-crash
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] Using cached minion ID from c:\salt\conf\minion_id: WIN-M9GCE6IBR1K.xyz.net
[DEBUG   ] Configuration file path: c:\salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged
.
[DEBUG   ] Reading configuration from c:\salt\conf\minion
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded jinja.render
[DEBUG   ] LazyLoaded yaml.render
[DEBUG   ] LazyLoaded state.highstate
[DEBUG   ] LazyLoaded grains.get
[DEBUG   ] LazyLoaded saltutil.is_running
[DEBUG   ] LazyLoaded roots.envs
[ERROR   ] Traceback (most recent call last):
  File "c:\python27\lib\site-packages\salt\scripts.py", line 335, in salt_call
    client.run()
  File "c:\python27\lib\site-packages\salt\cli\call.py", line 53, in run
    caller.run()
  File "c:\python27\lib\site-packages\salt\cli\caller.py", line 135, in run
    ret = self.call()
  File "c:\python27\lib\site-packages\salt\cli\caller.py", line 198, in call
    ret['return'] = func(*args, **kwargs)
  File "c:\python27\lib\site-packages\salt\modules\state.py", line 554, in highs
tate
    mocked=kwargs.get('mock', False))
  File "c:\python27\lib\site-packages\salt\state.py", line 3198, in __init__
    self.client = salt.fileclient.get_file_client(self.opts)
  File "c:\python27\lib\site-packages\salt\fileclient.py", line 56, in get_file_
client
    }.get(client, RemoteClient)(opts)
  File "c:\python27\lib\site-packages\salt\fileclient.py", line 1283, in __init_
_
    self.channel = salt.fileserver.FSChan(opts)
  File "c:\python27\lib\site-packages\salt\fileserver\__init__.py", line 723, in
 __init__
    self.fs.init()
  File "c:\python27\lib\site-packages\salt\fileserver\__init__.py", line 475, in
 init
    back = self._gen_back(back)
  File "c:\python27\lib\site-packages\salt\fileserver\__init__.py", line 345, in
 _gen_back
    if '{0}.envs'.format(sub) in self.servers:
  File "c:\python27\lib\_abcoll.py", line 388, in __contains__
    self[key]
  File "c:\python27\lib\site-packages\salt\loader.py", line 900, in __getitem__
    func = super(LazyLoader, self).__getitem__(item)
  File "c:\python27\lib\site-packages\salt\utils\lazy.py", line 89, in __getitem
__
    log.debug('LazyLoaded {0}'.format(key))
  File "c:\python27\lib\logging\__init__.py", line 1148, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "c:\python27\lib\site-packages\salt\log\setup.py", line 277, in _log
    self, level, msg, args, exc_info=exc_info, extra=extra
  File "c:\python27\lib\logging\__init__.py", line 1279, in _log
    self.handle(record)
  File "c:\python27\lib\logging\__init__.py", line 1289, in handle
    self.callHandlers(record)
  File "c:\python27\lib\logging\__init__.py", line 1329, in callHandlers
    hdlr.handle(record)
  File "c:\python27\lib\logging\__init__.py", line 757, in handle
    self.emit(record)
  File "c:\python27\lib\logging\__init__.py", line 870, in emit
    stream.write(ufs % msg)
KeyboardInterrupt

If I return from within the running() function without doing anything, the highstate call executes successfully. Here is the hack I'm currently working with to get my state files finished up.

salt/utils/minion.py:

def running(opts):
    '''
    Return the running jobs on this minion
    '''
    ret = []
    # HACK TO GET AROUND CTRL-C ISSUE FOR NOW
    return ret

    proc_dir = os.path.join(opts['cachedir'], 'proc')
    if not is.path.isdir(proc_dir):
        return ret
    for fn_ in ....
twangboy commented 8 years ago

@rambli Are you able to start/stop/disable/etc the Power service from the windows service manager? Are you able to issue a net stop power command? My windows machines all show this as not available for this specific service.

rambli commented 8 years ago

@twangboy This issue is not related to the power service, as I see the Ctrl-c issue even when running state.highstate with a very simple top.sls, like copy a file for example. If you could please read my previous comment, it has the issue described in detail :)

rambli commented 8 years ago

Hey guys, to eliminate any potential interference from my setup, I spun up a new Windows 2012 R2 VM and did a fresh install of Salt via pip (version 2015.8.7). I'm seeing the same issue :( Any ideas what could be going on?

twangboy commented 8 years ago

@rambli I didn't know you were installing salt using pip. Do you get the same issue if you install salt using the windows installer?

https://repo.saltstack.com/windows/Salt-Minion-2015.8.7-AMD64-Setup.exe

If you don't have problems with it using the windows installer, I would suspect something in your python environment. Try setting up a dev environment for salt using the dev_env.ps1 script at https://github.com/saltstack/salt-windows-dev. Be sure to checkout the 2015.8 branch. (Note: these scripts are moving to the salt repository for future versions of salt.)

rambli commented 8 years ago

@twangboy The windows installer worked fine for me! I tried Salt via pip on three machines (2 VMs, and one physical host), all three were clean installs, so unsure about the python environment being a problem. Please let me know if there's anything environment settings you'd like me to check on these.

twangboy commented 8 years ago

@rambli Did you try running the dev_env.ps1 script to install python instead of installing python manually?

deuscapturus commented 7 years ago

I had this same problem with one of my Windows 2012R2 minions. I had to uninstall, delete the entire C:\salt\ directory, and re-install to fix it.

PS C:\salt> .\salt-call.bat --versions-report
Salt Version:
           Salt: 2016.3.3

Dependency Versions:
           cffi: 1.6.0
       cherrypy: 5.3.0
       dateutil: 2.5.3
          gitdb: 0.6.4
      gitpython: 2.0.2
          ioflo: 1.5.3
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: 1.4.5
       M2Crypto: Not Installed
           Mako: 1.0.4
   msgpack-pure: Not Installed
 msgpack-python: 0.4.7
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) [MSC v.1500 64 bit (AMD64)]
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: 0.9.0
        timelib: 0.2.4
        Tornado: 4.3
            ZMQ: 4.1.2

System Versions:
           dist:
        machine: AMD64
        release: 8.1
         system: Windows
        version: 8.1 6.3.9600  Multiprocessor Free
DrMerlin commented 7 years ago

Me too!

stale[bot] commented 6 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.