Workers terminated by SigKill 9

[byang223]

I have an endpoint that does the following

def some_api_request:
  proc = subprocess.Popen()
  output, errs = proc.communicate()
  // do some other things and then respond 200 with payload built from output

the original worker (process A) spawns a child process (process B). Process B successfully runs to completion, but during that time, process A gets terminated by Sigkill with the following message in logs. this seems to happen randomly (most of the time the request works)

 DAMN ! worker 8 (pid: 143110) died, killed by signal 9 :( trying respawn ...

I'm trying to determine why the worker is getting killed but there's no additional logging. I see in the code there are a bunch of "uwsgi_log_verbose" lines but I'm not sure how to enable/access those logs.

I also see recommendations that people should be using "close-on-exec" option when spawning subprocesses but I don't really understand why this may or may not be an issue. I'm concerned that "close-on-exec" would kill the connection upstream but I want the subprocess to complete and use the data returned to respond to the original request. I'm not sure what other implications this flag can have, but FWIW I'm not seeing any file descriptor leaks in my application.

I'm currently suspecting "reload-on-rss" to be the cause due to potential memory leaks. Is there any way to verify this in the logs? My hesitation is I was under the impression that when the threshold gets crossed uwsgi prevents the worker from handling anymore requests in the future. Is it possible that after the threshold is crossed, the worker gets forcibly reloaded in the middle of a future request?

Configuration file is below

{
  "uwsgi": {
    "chmod-socket": 666,
    "logfile-chmod": 755,
    "pidfile": redacted,
    "socket": redacted,
    "stats": redacted,
    "plugins": "python",
    "autoload": true,
    "master": true,
    "no-orphans": true,
    "memory-report": true,
    "enable-threads": true,
    "log-micros": true,
    "log-date": true,
    "die-on-term": true,
    "disable-logging": true,
    "logto": redacted,
    "listen": 100,
    "buffer-size": 32768,
    "module": "clay.wsgi",
    "callable": "application",
    "reload-on-rss": 3000,
    "workers": 8,
    "ignore-write-errors": false,
    "disable-write-exception": false,
    "thunder-lock": true
  }
}