search

ExchangeUnion / xud

Exchange Union Daemon 🔁 ⚡️
https://exchangeunion.com
GNU Affero General Public License v3.0
115 stars 44 forks source link

xud sometimes not stopping gracefully via SIGTERM or shutdown #1278

Closed kilrau closed 3 years ago

kilrau commented 5 years ago

Discovered on master pre 1.0.0-mainnet tag by @offerm 

admin@lightning:~ $ sudo systemctl start xud
admin@lightning:~ $ sudo systemctl stop xud
admin@lightning:~ $ sudo systemctl status xud
● xud.service - xud Daemon
   Loaded: loaded (/etc/systemd/system/xud.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-10-03 16:57:26 -03; 3s ago
  Process: 30032 ExecStart=/bin/bash /home/bitcoin/scripts/run_xud (code=killed, signal=TERM)
 Main PID: 30032 (code=killed, signal=TERM)
    Tasks: 12 (limit: 2319)
   Memory: 53.1M
   CGroup: /system.slice/xud.service
           └─30033 node /home/bitcoin/github/xud/bin/xud --mainnet --lnd.LTC.disable

Oct 03 16:57:22 lightning systemd[1]: Started xud Daemon.
Oct 03 16:57:26 lightning systemd[1]: Stopping xud Daemon...
Oct 03 16:57:26 lightning systemd[1]: xud.service: Main process exited, code=killed, status=15/TERM
Oct 03 16:57:26 lightning systemd[1]: xud.service: Succeeded.
Oct 03 16:57:26 lightning systemd[1]: Stopped xud Daemon.
admin@lightning:~ $ ps -ef |grep xud
bitcoin  30033     1 61 16:57 ?        00:00:07 node /home/bitcoin/github/xud/bin/xud --mainnet --lnd.LTC.disable
admin    30074 28920  0 16:57 pts/1    00:00:00 grep --color=auto xud

Please confirm that this is still the case on latest master @offerm , then @sangaman can take over.

michael1011 commented 5 years ago

@offerm what does your xud.service file look like? Did you use the one from the docs?

kilrau commented 4 years ago

To be done: test manual xud setup with sample service fail and test if it starts/stops properly.

engwarrior commented 4 years ago

Pls provide estimate for this task.

sangaman commented 4 years ago

There are still a handful of scenarios where xud doesn't shutdown gracefully given a sigterm or shutdown command, I've tried a couple of times to diagnose what the issue is but with no success so far. I don't think it's related to systemd necessarily, so I will rename this issue and use it to track general shutdown issues.

kilrau commented 4 years ago

I found one more: when xud is Waiting for lndbtc admin.macaroon or lnd-BTC is WaitingUnlock it doesn't react to shutdown/sigterm:

29/04/2020 11:10:20.002 [LND-BTC] error: getinfo error: 4.2 - lnd-BTC is WaitingUnlock
29/04/2020 11:10:20.003 [LND-LTC] error: getinfo error: 4.2 - lnd-LTC is WaitingUnlock
29/04/2020 11:10:30.035 [RPC] debug: received call /xudrpc.Xud/GetBalance
29/04/2020 11:10:31.240 [GLOBAL] info: XUD is shutting down
29/04/2020 11:10:31.242 [LND-BTC] info: LndClient status: Disconnected
29/04/2020 11:10:31.243 [LND-LTC] info: LndClient status: Disconnected
29/04/2020 11:10:31.243 [CONNEXT] info: ConnextClient status: Disconnected
29/04/2020 11:10:31.245 [P2P] debug: Peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding): closing socket. reason: Shutdown
29/04/2020 11:10:31.246 [P2P] debug: Peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden): closing socket. reason: Shutdown
29/04/2020 11:10:31.247 [P2P] debug: Peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift): closing socket. reason: Shutdown
29/04/2020 11:10:31.248 [P2P] debug: Peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse): closing socket. reason: Shutdown
29/04/2020 11:10:31.249 [P2P] debug: Peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind): closing socket. reason: Shutdown
29/04/2020 11:10:31.249 [P2P] debug: Peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch): closing socket. reason: Shutdown
29/04/2020 11:10:31.251 [HTTP] info: http server has closed
29/04/2020 11:10:31.252 [RPC] info: gRPC server completed shutdown
29/04/2020 11:10:31.253 [ORDERBOOK] debug: removed all orders for peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding)
29/04/2020 11:10:31.253 [ORDERBOOK] debug: removed all orders for peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden)
29/04/2020 11:10:31.254 [ORDERBOOK] debug: removed all orders for peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift)
29/04/2020 11:10:31.255 [ORDERBOOK] debug: removed all orders for peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse)
29/04/2020 11:10:31.255 [ORDERBOOK] debug: removed all orders for peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind)
29/04/2020 11:10:31.256 [ORDERBOOK] debug: removed all orders for peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch)
29/04/2020 11:10:31.257 [GLOBAL] info: XUD shutdown gracefully
29/04/2020 11:10:31.258 [P2P] info: Peer 02e8408be3392249f5caf7542598cc0d070f9c976b8d2a127549d87e6816830e34 (HockeyPunch) socket closed
29/04/2020 11:10:31.258 [P2P] info: Peer 030a33024f634bccd5ca781928c9bcb03d9b0b2ca2a9014f9d7ef717ffda812479 (AcousticRemind) socket closed
29/04/2020 11:10:31.258 [P2P] info: Peer 02516c6b0e9d37ed130b0b5a13251b828759393ebb79e19e4479e3a4fbe7ad9599 (TapeUniverse) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 02514021570812ff0ad137698de74bbe4665869765f9c4758f8470d5f57a4e9044 (ExampleShift) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 02d532758864c49dd3ba9715d5e9e4d10c37f0fdc52b37c26e34c226ce8d548001 (PhraseHidden) socket closed
29/04/2020 11:10:31.259 [P2P] info: Peer 03193508d52dc8235c1bf6e902dae30c6dc1823f1e3cdc3c1a97652af9142cd4fe (MeltWedding) socket closed
kilrau commented 4 years ago

Can you help testing if in above two scenarios (easy reproducable with a new xud-docker environment, just don't "create") xud is still not shutting down gracefully? If it works now, we can close here. @raladev

kilrau commented 3 years ago

Closing here in favor of https://github.com/ExchangeUnion/xud/issues/1668