Closed frogunder closed 6 years ago
This has been happening for a long time and it's not specific to debug mode. If you look in the master log when you stop the master, you'll see this happens all the time.
It's happening because of the shenanigans done to juggle signal handlers in the process module (https://github.com/saltstack/salt/blob/develop/salt/utils/process.py#L697). Basically every process spawned gets assigned SIG_DFL, which for SIGINT, SIGTERM, and SIGQUIT, terminates the process. Furthermore, the master calls os.setsid()
when it daemonizes, making itself and all its children members of a process group, meaning any signal sent to the parent process will be propagated to all children. The end result is the children die before the master can clean up.
The fix for this would be to assign a custom handler (to either ignore signals, to communicate receipt of a signal to the parent process, do some cleanup, or a combination thereof).
We did have this happening for a while but I can't replicate this against the 2016.3.3 tag at all. Is this happening consistently or just on occasion?
It's consistent in 2016.3.2. It's also pretty easy to reproduce, just fire up the master in the foreground and then ^C it.
I can give 2016.3.3 a shot myself and let you know what happens.
I just tested against a fresh 2016.3.3 install in Vagrant on Ubuntu 14.04. Still happens.
^C[DEBUG ] ZeroMQReqServerChannel received a SIGINT. Exiting
[DEBUG ] ZeroMQReqServerChannel received a SIGINT. Exiting
[DEBUG ] EventPublisher received a SIGINT. Exiting
[DEBUG ] Maintenance received a SIGINT. Exiting
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 862, in emit
[DEBUG ] MWorker received a SIGINT. Exiting
[DEBUG ] MWorker received a SIGINT. Exiting
[DEBUG ] MWorker received a SIGINT. Exiting
[DEBUG ] MWorker received a SIGINT. Exiting
Traceback (most recent call last):
File "/usr/lib/python2.7/logging/__init__.py", line 862, in emit
[DEBUG ] Reactor received a SIGINT. Exiting
stream.write(ufs % msg)
stream.write(ufs % msg)
IOError: [Errno 0] Error
Logged from file process.py, line 657
IOError: [Errno 0] Error
Logged from file process.py, line 657
[INFO ] Some processes failed to respect the KILL signal: Process: <Process(ReqServer, started)> (Pid: 8958)
[INFO ] kill_children retries left: 3
[DEBUG ] Reactor received a SIGTERM. Exiting
[WARNING ] Master received a SIGINT. Exiting.
[INFO ] The Salt Master is shut down
The salt master is shutdown. Master received a SIGINT. Exited.Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/multiprocessing/util.py", line 321, in _exit_function
p._popen.terminate()
File "/usr/lib/python2.7/multiprocessing/forking.py", line 171, in terminate
os.kill(self.pid, signal.SIGTERM)
OSError: [Errno 3] No such process
Error in sys.exitfunc:
[ERROR ] An un-handled exception was caught by salt's global exception handler:
OSError: [Errno 3] No such process
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/multiprocessing/util.py", line 321, in _exit_function
p._popen.terminate()
File "/usr/lib/python2.7/multiprocessing/forking.py", line 171, in terminate
os.kill(self.pid, signal.SIGTERM)
OSError: [Errno 3] No such process
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.7/multiprocessing/util.py", line 321, in _exit_function
p._popen.terminate()
File "/usr/lib/python2.7/multiprocessing/forking.py", line 171, in terminate
os.kill(self.pid, signal.SIGTERM)
OSError: [Errno 3] No such process
You can see pretty clearly that SIGINT is propagating to child processes.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.
Description of Issue/Question
When running the master in debug mode and exit, it gives a Traceback when exiting.
Setup
(Please provide relevant configs and/or SLS files (Be sure to remove sensitive info).)
Steps to Reproduce Issue
Run the master in debug mode
salt-master -ldebug
Exit out of itCTRL-C
Versions Report