siad doesn't stop when shutting down/rebooting

[timrettop]

I've been having trouble lately with reboots/shutdowns and have traced it down to siad not quitting and wondering if it something I should seek support from here or siafoundation.

Ubuntu 20.04.3. I think it started after updating to 0.5.1 of handyhost.

If I stop the handyhost service, it doesn't stop said either. seems only SIGKILL before shutdown/reboot is needed currently.

This is a section of the handyhost.log, not sure if the info here is helpful. I believe its from 2 shutdown/reboots where kill -9 was used, and the last section is my attempt to shutdown handyhost service (but siad is still running).

rpcPort :9981
muxPort :9983
muxWSPort :9984
apiPort :10801
tryingwallet unlock
status not 200 204 No Content
no json response
wallet unlock success
init SC health check interval
error: Forever detected script was killed by signal: SIGKILL
(node:2000) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
HandyHost Daemon Running at: http://10.1.50.15:8008/, and https://10.1.50.15:58008/ (self-signed cert)
no version, must be dead
start siad /root/.HandyHost/siaData
hostPort :9982
rpcPort :9981
muxPort :9983
muxWSPort :9984
apiPort :10801
tryingwallet unlock
status not 200 490 status code 490
status code is 490
error unlocking wallet 490
tryingwallet unlock
status not 200 204 No Content
no json response
wallet unlock success
init SC health check interval
performing SC health check
SC is alive
performing SC health check
SC is alive
error: Forever detected script was killed by signal: SIGKILL
(node:2028) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
(Use `node --trace-warnings ...` to show where the warning was created)
HandyHost Daemon Running at: http://10.1.50.15:8008/, and https://10.1.50.15:58008/ (self-signed cert)
no version, must be dead
start siad /root/.HandyHost/siaData
hostPort :9982
rpcPort :9981
muxPort :9983
muxWSPort :9984
apiPort :10801
tryingwallet unlock
status not 200 490 status code 490
status code is 490
error unlocking wallet 490
tryingwallet unlock
status not 200 204 No Content
no json response
wallet unlock success
init SC health check interval
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
performing SC health check
SC is alive
is positional [ 'api', 'sia', 'getStorage' ]
is positional [ 'api', 'sia', 'getHostPublicKey' ]
is positional [ 'api', 'sia', 'getHostMetrics' ]
is positional [ 'api', 'sia', 'getContracts' ]
is positional [ 'api', 'getIP' ]
is positional [ 'api', 'sia', 'getPorts' ]
is positional [ 'api', 'sia', 'getHostConfig' ]
is positional [ 'api', 'sia', 'getChainStatus' ]
is positional [ 'api', 'sia', 'getChainStatus' ]
is positional [ 'api', 'sia', 'getPorts' ]
error: Forever detected script was killed by signal: SIGKILL

Let me know if I can provide any other details to help diagnose.

[timrettop]

There are no logs shown in any of the siad services logs when the handyhost service is stopped.

This is the output of service status after stopping handyhost.

tim@black:~$ sudo service handyhost status
● handyhost.service - LSB: HandyHost
     Loaded: loaded (/etc/init.d/handyhost; generated)
     Active: inactive (dead) since Sun 2022-01-02 20:54:12 PST; 4s ago
       Docs: man:systemd-sysv-generator(8)
    Process: 294126 ExecStart=/etc/init.d/handyhost start (code=exited, status=0/SUCCESS)
    Process: 297873 ExecStop=/etc/init.d/handyhost stop (code=exited, status=0/SUCCESS)
      Tasks: 15 (limit: 38265)
     Memory: 7.0G
     CGroup: /system.slice/handyhost.service
             ├─  2099 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             ├─  2100 /usr/bin/dbus-daemon --syslog --fork --print-pid 4 --print-address 6 --session
             └─294641 siad --api-addr localhost:10801 --siamux-addr :9983 --siamux-addr-ws :9984 --host-addr :9982 --rpc-addr :9981 --sia-directory /root/.HandyHost/>

Jan 02 20:54:11 black handyhost[297873]: Shutting down handyhost node instance : Found '/opt/handyhost/.nvmrc' with version <14.16.1>
Jan 02 20:54:11 black handyhost[297873]: Now using node v14.16.1 (npm v6.14.12)
Jan 02 20:54:12 black handyhost[298314]: (node:298314) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
Jan 02 20:54:12 black handyhost[298314]: (Use `node --trace-warnings ...` to show where the warning was created)
Jan 02 20:54:12 black handyhost[298314]: (node:298314) Warning: Accessing non-existent property 'padLevels' of module exports inside circular dependency
Jan 02 20:54:12 black handyhost[298314]: info:    Forever stopped process:
Jan 02 20:54:12 black handyhost[298314]:     uid  command                                    script forever pid    id logfile                uptime
Jan 02 20:54:12 black handyhost[298314]: [0] NSlb /root/.nvm/versions/node/v14.16.1/bin/node app.js 294600  294611    /var/log/handyhost.log 0:0:4:35.56
Jan 02 20:54:12 black systemd[1]: handyhost.service: Succeeded.
Jan 02 20:54:12 black systemd[1]: Stopped LSB: HandyHost.

[timrettop]

sigquit on the process ID successfully stops the service

[alexsmith540]

So keeping siad alive is generally expected behavior. When you're hosting, you want siad to stay alive always else you risk losing your collateral. Hence we do a bunch of tasks to keep sia alive always. If you're not actively hosting with sia and want it to not be on, you can: rm -rf /root/.HandyHost/siaData which will remove the chain data and not startup siad on app start.

[timrettop]

So keeping siad alive is generally expected behavior. When you're hosting, you want siad to stay alive always else you risk losing your collateral. Hence we do a bunch of tasks to keep sia alive always. If you're not actively hosting with sia and want it to not be on, you can: rm -rf /root/.HandyHost/siaData which will remove the chain data and not startup siad on app start.

Sorry for the slow response, I missed your follow up. The concern I have is that the process hangs up the restart process on my server, leaving it in a hung state. How do I ensure siad stops when I issue a reboot?

[alexsmith540]

Ah I see. So to issue a clean shutdown of siad you can run the following command: siac -a localhost:10801 stop and it may take up to a minute for siad to shutdown/stop completely.

Also, most of the app is owned by root so likely you'll need to run this as root, and make sure to source the siac binary path which can be done like source ~/.profile Hope that helps.

[timrettop]

As root I ran source ~/.profile then ran siac -a localhost:10801 stop It outputted that siad stopped, however the process doesn't stop, even after 5 minutes.

[alexsmith540]

ouch, yeah seems like probably zombie siad instance is hanging then. Usually when I get a zombie siad I have to pkill -9 siad or do a hard reboot on the hardware to resolve. Not an easy way around this one but it's possible that somebody in the Sia community knows some cleaner way to kill a zombie siad. Sounds super annoying but if it were me doing some pre-shutdown script I'd probably try the clean stop with siac, and then set a timeout for a few minutes and pkill -9 siad. Not convenient at all though :(

[timrettop]

I would have guessed that siad should be halted when the node forever stop is called, but it seems that the child process is unreferenced during the spawning.

There is a haltSiad function available which seems to be used for updates to sia and maybe for port changes? Could this be incorporated into a 'teardown' when forever stop is called? I'm a novice with Node, so I'm just spitballing.

My other thought was to try to clean up via the debian init script, but I don't think that would safely manage the siad process started by node, unless usign a pid file maybe.