request help: there are two go-runner processes after killing the privileged agent process

[zdzh]

Issue description

我希望在apisix服务中达成的一个效果是kill掉任何一个子进程时apisix服务都能重新拉起一个子进程。 实际测试发现kill掉privileged agent process后，apisix服务确实重新拉起了新的privileged agent process，但却出现了两个apisix-go-runner，而kill掉其他子进程均能正常拉起，,这点是故意为之还是有方法可以保证privileged agent process被拉起后只存在一个go-runner进程。

$ systemctl status apisix
● apisix.service - apisix
   Loaded: loaded (/usr/lib/systemd/system/apisix.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-22 09:44:14 +08; 3min 53s ago
  Process: 19804 ExecStop=/usr/bin/apisix stop (code=exited, status=0/SUCCESS)
  Process: 20041 ExecStartPost=/sf/bin/apisix-config.py (code=exited, status=0/SUCCESS)
  Process: 19910 ExecStart=/usr/bin/apisix start (code=exited, status=0/SUCCESS)
  Process: 19907 ExecStartPre=/usr/bin/ln -sf /usr/local/openresty/nginx/sbin/nginx /usr/bin/openresty (code=exited, status=0/SUCCESS)
  Process: 19900 ExecStartPre=/usr/bin/bash -c if [ ! -f /etc/resolv.conf -o  `cat /etc/resolv.conf | wc -l` -eq 0 ]; then /usr/bin/echo 'nameserver 114.114.114.114' > /etc/resolv.conf; fi (code=exited, status=0/SUCCESS)
 Main PID: 20029 (openresty)
   Memory: 27.7M
   CGroup: /system.slice/apisix.service
           ├─20029 nginx: master process openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf
           ├─20030 nginx: worker process
           ├─20032 nginx: cache manager process
           ├─20034 nginx: privileged agent process
           └─20080 /sf/bin/apisix-go-runner run

Oct 22 09:44:13 aops-fefcfe677336 systemd[1]: Starting apisix...
Oct 22 09:44:13 aops-fefcfe677336 apisix[19910]: /usr/local/openresty/luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua start
Oct 22 09:44:13 aops-fefcfe677336 apisix[19910]: Warning! Current maximum number of open file descriptors [1024] is too small, please increase user limits by execute ...nce is low.Oct 22 09:44:14 aops-fefcfe677336 systemd[1]: Started apisix.
Hint: Some lines were ellipsized, use -l to show in full.

## kill privileged agent process
$  kill -9 20034

## after kill privileged agent process
$  systemctl status apisix
● apisix.service - apisix
   Loaded: loaded (/usr/lib/systemd/system/apisix.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2021-10-22 09:44:14 +08; 15min ago
  Process: 19804 ExecStop=/usr/bin/apisix stop (code=exited, status=0/SUCCESS)
  Process: 20041 ExecStartPost=/sf/bin/apisix-config.py (code=exited, status=0/SUCCESS)
  Process: 19910 ExecStart=/usr/bin/apisix start (code=exited, status=0/SUCCESS)
  Process: 19907 ExecStartPre=/usr/bin/ln -sf /usr/local/openresty/nginx/sbin/nginx /usr/bin/openresty (code=exited, status=0/SUCCESS)
  Process: 19900 ExecStartPre=/usr/bin/bash -c if [ ! -f /etc/resolv.conf -o  `cat /etc/resolv.conf | wc -l` -eq 0 ]; then /usr/bin/echo 'nameserver 114.114.114.114' > /etc/resolv.conf; fi (code=exited, status=0/SUCCESS)
 Main PID: 20029 (openresty)
   Memory: 30.8M
   CGroup: /system.slice/apisix.service
           ├─ 20029 nginx: master process openresty -p /usr/local/apisix -c /usr/local/apisix/conf/nginx.conf
           ├─ 20030 nginx: worker process
           ├─ 20080 /sf/bin/apisix-go-runner run     # 原先的go-runner进程依然存在
           ├─ 57599 nginx: cache manager process
           ├─150875 nginx: privileged agent process
           └─150884 /sf/bin/apisix-go-runner run   # 新拉起了go-runner进程

Oct 22 09:44:13 aops-fefcfe677336 systemd[1]: Starting apisix...
Oct 22 09:44:13 aops-fefcfe677336 apisix[19910]: /usr/local/openresty/luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua start
Oct 22 09:44:13 aops-fefcfe677336 apisix[19910]: Warning! Current maximum number of open file descriptors [1024] is too small, please increase user limits by execute ...nce is low.Oct 22 09:44:14 aops-fefcfe677336 systemd[1]: Started apisix.
Hint: Some lines were ellipsized, use -l to show in full.

Environment

• apisix version (cmd: apisix version): 2.10.0
• OS (cmd: uname -a):
• OpenResty / Nginx version (cmd: nginx -V or openresty -V): openresty/1.19.3.1
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have:
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):

[spacewander]

The Go runner is managed by the privileged agent. If the agent is killed, there is no simple way to know if the Go runner is still alive. There is complex way to do it, but as the crash of privileged agent is rare so I don't think it is worth to do it.

[spacewander]

We can leave this issue open and see if it is worth handling it.

[zdzh]

thanks

[github-actions[bot]]

This issue has been marked as stale due to 350 days of inactivity. It will be closed in 2 weeks if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

[github-actions[bot]]

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.