search

fort-nix / nix-bitcoin

A collection of Nix packages and NixOS modules for easily installing full-featured Bitcoin nodes with an emphasis on security.
https://nixbitcoin.org
MIT License
492 stars 101 forks source link

Service dependency failed after the system reboot #595

Open seberm opened 1 year ago

seberm commented 1 year ago

Hello, when I reboot the VM with nix-bitcoin, most of the services which depend on a bitcoind.service fail with the "Dependency failed for 'service_name'" error and they are not started/restarted at all.

It's a pain especially when the clightning.service is not started because most of other services depend on it:

$ systemctl status clightning.service
○ clightning.service
     Loaded: loaded (/etc/systemd/system/clightning.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for clightning.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: clightning.service: Job clightning.service/start failed with result 'dependency'.

$ systemctl status btcpayserver.service
○ btcpayserver.service
     Loaded: loaded (/etc/systemd/system/btcpayserver.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for btcpayserver.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: btcpayserver.service: Job btcpayserver.service/start failed with result 'dependency'.
$ systemctl status mempool.service
○ mempool.service
     Loaded: loaded (/etc/systemd/system/mempool.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for mempool.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: mempool.service: Job mempool.service/start failed with result 'dependency'.

$ systemctl status electrs.service
○ electrs.service
     Loaded: loaded (/etc/systemd/system/electrs.service; enabled; preset: enabled)
     Active: inactive (dead)

Mar 03 13:53:27 nix-bitcoin systemd[1]: Dependency failed for electrs.service.
Mar 03 13:53:27 nix-bitcoin systemd[1]: electrs.service: Job electrs.service/start failed with result 'dependency'
$ nodeinfo
{
  "bitcoind": {
    "local_address": "127.0.0.1:8333",
    "onion_address": "addr.onion:8333"
  },
  "btcpayserver": "'btcpayserver.service' is not running",
  "clightning": "'clightning.service' is not running",
  "clightning-rest": "'clightning-rest.service' is not running",
  "electrs": "'electrs.service' is not running",
  "mempool": "'mempool.service' is not running",
  "mempool-frontend": {
    "local_address": "127.0.0.1:60845"
  },
  "rtl": "'rtl.service' is not running",
  "sshd": {
    "onion_address": "addr.onion:22"
  }
}

This behavior is reproducible in 90% of reboots. I am running latest nix-bitcoin (v0.0.87) on:

$ cat /etc/os-release
BUILD_ID="22.11pre-git"
PRETTY_NAME="NixOS 22.11 (Raccoon)"
VERSION="22.11 (Raccoon)"
...

I think it could be possible that it just takes too long to start bitcoind unit and initialize the Tor network which results in dependent units timing out. What do you think?

My current workaround is just to manually restart of all the failed units.

jonasnick commented 1 year ago

Can you see in the bitcoind logs that it takes long to start? Are there any errors in bitcoind logs?

seberm commented 1 year ago

Hello Jonas, I finally found the problem which is causing this issue.

The problem:

1) I run krops-deploy, which does rsync of my secrets to a remote node with my local permissions. These are:

$ ls -alh /var/src/secrets/
total 80K
drwx------ 2 root       root              4.0K Mar  3 19:38 .
drwxr-xr-x 6 root       root              4.0K Mar  3 13:48 ..
-rw------- 1 root       root                32 Mar  3 19:23 backup-encryption-env
-rw------- 1 root       root                21 Mar  3 19:23 backup-encryption-password
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-btcpayserver
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-privileged
-rw-r--r-- 1 bitcoin    bitcoin-exporter    98 Dec 30 14:32 bitcoin-HMAC-prometheus
-rw-r--r-- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-public
-rw-r--r-- 1 bitcoin    nbxplorer           21 Aug 24  2022 bitcoin-rpcpassword-btcpayserver
-rw-r--r-- 1 bitcoin    bitcoin             21 Aug 24  2022 bitcoin-rpcpassword-privileged
-rw-r--r-- 1 bitcoin    bitcoin-exporter    21 Dec 30 12:38 bitcoin-rpcpassword-prometheus
-rw-r--r-- 1 bitcoin    bitcoinrpc-public   21 Aug 24  2022 bitcoin-rpcpassword-public
-rw------- 1 clightning clightning          21 Mar  3 19:38 clightning-replication-password
-rw------- 1 nginx      nginx              692 Nov 15 13:17 nginx-btcpayserver-cert
-rw-r--r-- 1 root       root                 1 Nov 15 13:17 nginx-btcpayserver-cert-alt-names
-rw------- 1 nginx      nginx              241 Nov 15 13:17 nginx-btcpayserver-key
-rw-r--r-- 1 root       root               662 Dec 26 22:49 README.md
-rw-r--r-- 1 root       root                26 Jan 31 23:04 root_password
-rw-r--r-- 1 rtl        rtl                 21 Aug 23  2022 rtl-password
-rw-r--r-- 1 root       root               241 Dec 25 20:18 tor-bridges

2) Because I do stop all of the services before the deployment using krops writeCommand, I need to reboot the remote machine.

krops.pkgs.krops.writeCommand "deploy-boot" {
  inherit target;
  source = import ./sources.nix { inherit extraSources krops; };

  command = targetPath: ''
    systemctl stop btcpayserver
    systemctl stop mempool
    systemctl stop electrs

    systemctl stop clightning
    systemctl stop bitcoind

    sync

    nixos-rebuild -I ${targetPath} boot
  '';
...
}

3) Machine is rebooted and you were correct, there are erros in journalctl -u bitcoind:

Mar 04 22:01:39 nix-bitcoin systemd[1]: Starting bitcoind.service...
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1003]: cat: /var/src/secrets/bitcoin-HMAC-btcpayserver: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1004]: cat: /var/src/secrets/bitcoin-HMAC-privileged: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1005]: cat: /var/src/secrets/bitcoin-HMAC-prometheus: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind-pre-start[1006]: cat: /var/src/secrets/bitcoin-HMAC-public: Permission denied
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: 2023-03-04T21:01:39Z Ignoring unknown configuration value rpcconnect
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Bitcoin Core version v24.0.1 (release build)
...
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Using rpcauth authentication.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Invalid -rpcauth argument.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Error: Unable to start HTTP server. See debug log for details.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Error: Unable to start HTTP server. See debug log for details.
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Shutdown: In progress...
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: scheduler thread exit
Mar 04 22:01:39 nix-bitcoin bitcoind[1010]: Shutdown: done
Mar 04 22:01:39 nix-bitcoin systemd[1]: bitcoind.service: Main process exited, code=exited, status=1/FAILURE
Mar 04 22:01:39 nix-bitcoin systemd[1]: bitcoind.service: Failed with result 'exit-code'.
Mar 04 22:01:39 nix-bitcoin systemd[1]: Failed to start bitcoind.service.

That is why all the dependent units fail. These errors are caused because the setup-secrets.service probably hasn't finished before the bitcoind was started.

The reason I have overlooked the error is because the bitcoind service is automatically restarted and then it's running without any errors.

It's running without any issues, because the setup-secrets.service unit has finished meanwhile.

Then the permissions are set to:

$ ls -alh /var/src/secrets/
total 80K
drwxr-x--x 2 root       root              4.0K Mar  3 19:38 .
drwxr-xr-x 6 root       root              4.0K Mar  3 13:48 ..
-r--r----- 1 root       root                32 Mar  3 19:23 backup-encryption-env
-r--r----- 1 root       root                21 Mar  3 19:23 backup-encryption-password
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-btcpayserver
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-privileged
-r--r----- 1 bitcoin    bitcoin-exporter    98 Dec 30 14:32 bitcoin-HMAC-prometheus
-r--r----- 1 bitcoin    bitcoin             98 Aug 25  2022 bitcoin-HMAC-public
-r--r----- 1 bitcoin    nbxplorer           21 Aug 24  2022 bitcoin-rpcpassword-btcpayserver
-r--r----- 1 bitcoin    bitcoin             21 Aug 24  2022 bitcoin-rpcpassword-privileged
-r--r----- 1 bitcoin    bitcoin-exporter    21 Dec 30 12:38 bitcoin-rpcpassword-prometheus
-r--r----- 1 bitcoin    bitcoinrpc-public   21 Aug 24  2022 bitcoin-rpcpassword-public
-r--r----- 1 clightning clightning          21 Mar  3 19:38 clightning-replication-password
-r--r----- 1 nginx      nginx              692 Nov 15 13:17 nginx-btcpayserver-cert
-r--r----- 1 root       root                 1 Nov 15 13:17 nginx-btcpayserver-cert-alt-names
-r--r----- 1 nginx      nginx              241 Nov 15 13:17 nginx-btcpayserver-key
-r--r----- 1 root       root               662 Dec 26 22:49 README.md
-r--r----- 1 root       root                26 Jan 31 23:04 root_password
-r--r----- 1 rtl        rtl                 21 Aug 23  2022 rtl-password
-r--r----- 1 root       root               241 Dec 25 20:18 tor-bridges

Workaround

Is it possible that bitcoind is started before the setup-secrets.service/nix-bitcoin-secrets.target hasn't finished?

My current workaround is just to restart the setup-secrets.service right after the deployment in the krops writeCommand, so the correct permissions are set before the machine reboots.

Thanks!

elsirion commented 1 year ago

I have a similar problem: bitcoind crashes from time to time (unreliable fs cache, need to fix) and clightning stops but starts again, but clightning-rest, rtl and spark-wallet stay in the "dependency failed" state.

Systemd output `bitcoind` journal around the crash time: ``` May 10 11:45:43 btc-2 bitcoind[1555235]: ERROR: ReadRawBlockFromDisk: Block magic mismatch for FlatFilePos(nFile=3380, nPos=53835290): 00000000 versus expected f9beb4d9 May 10 11:45:43 btc-2 bitcoind[1555235]: bitcoind: net_processing.cpp:2149: void {anonymous}::PeerManagerImpl::ProcessGetBlockData(CNode&, {anonymous}::Peer&, const CInv&): Assertion `!"cannot load block from disk"' failed. May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Main process exited, code=killed, status=6/ABRT May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Failed with result 'signal'. May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Consumed 5min 8.276s CPU time, received 104.5M IP traffic, sent 2.9G IP traffic. May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Scheduled restart job, restart counter is at 2. May 10 11:45:51 btc-2 systemd[1]: Stopped bitcoind.service. May 10 11:45:51 btc-2 systemd[1]: bitcoind.service: Consumed 5min 8.276s CPU time, received 104.5M IP traffic, sent 2.9G IP traffic. May 10 11:45:51 btc-2 systemd[1]: Starting bitcoind.service... May 10 11:45:52 btc-2 bitcoind[1557879]: 2023-05-10T11:45:52Z Ignoring unknown configuration value rpcconnect May 10 11:45:52 btc-2 bitcoind[1557879]: Bitcoin Core version v24.0.1 (release build) May 10 11:45:52 btc-2 bitcoind[1557879]: Using the 'x86_shani(1way,2way)' SHA256 implementation May 10 11:45:52 btc-2 bitcoind[1557879]: Using RdSeed as an additional entropy source … May 10 11:46:05 btc-2 bitcoind[1557879]: net thread start May 10 11:46:05 btc-2 bitcoind[1557879]: addcon thread start May 10 11:46:05 btc-2 bitcoind[1557879]: msghand thread start May 10 11:46:05 btc-2 bitcoind[1557879]: opencon thread start May 10 11:46:05 btc-2 bitcoind[1557879]: dnsseed thread start May 10 11:46:05 btc-2 bitcoind[1557879]: Waiting 300 seconds before querying DNS seeds. May 10 11:46:05 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000022f5c52126c931c9d504574be31236d57293ea8567783 height=788975 version=0x2fffe000 log2_work=94.169366 tx=834958440 date='2023-05-09T20:04:39Z' progress=0.999802 cache=1.8MiB(12777txo) May 10 11:46:05 btc-2 bitcoind[1557879]: init message: Done loading May 10 11:46:05 btc-2 systemd[1]: Started bitcoind.service. May 10 11:46:07 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000005b938bff437192403c98b4910aebd18950441313617df height=788976 version=0x20002000 log2_work=94.169379 tx=834961732 date='2023-05-09T20:07:20Z' progress=0.999803 cache=3.4MiB(23093txo) … May 10 11:46:57 btc-2 bitcoind[1557879]: Leaving InitialBlockDownload (latching to false) May 10 11:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000023ac6d1e121cdedd5fb54fb7a69b43511f0150466ab86 height=789066 version=0x3fff0000 log2_work=94.170581 tx=835327909 date='2023-05-10T11:21:14Z' progress=0.999995 cache=116.4MiB(765810txo) May 10 11:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000002fe2e6894f341e36856b3c01ade1a467d43682493a0ab height=789067 version=0x20c00000 log2_work=94.170594 tx=835332576 date='2023-05-10T11:21:43Z' progress=0.999995 cache=117.3MiB(772839txo) May 10 11:47:04 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000002c6be865bd1f67df8935bff5e6c325a8dd7d1ead0665c height=789068 version=0x26bfe000 log2_work=94.170607 tx=835336991 date='2023-05-10T11:22:16Z' progress=0.999995 cache=118.5MiB(781703txo) May 10 11:47:04 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000055c85d5f78ad3e08a5bf825a56d0a756a476b475e1fa4 height=789069 version=0x25ce4000 log2_work=94.170621 tx=835341002 date='2023-05-10T11:23:34Z' progress=0.999995 cache=119.9MiB(792108txo) May 10 11:47:10 btc-2 bitcoind[1557879]: Imported mempool transactions from disk: 8584 succeeded, 10732 failed, 0 expired, 0 already there, 0 waiting for initial broadcast May 10 11:47:10 btc-2 bitcoind[1557879]: loadblk thread exit ``` `clightning` journal around the crash time: ``` May 10 11:45:51 btc-2 lightningd[1556215]: UNUSUAL plugin-bcli: bitcoin-cli -datadir=/bitcoin -rpcconnect=127.0.0.1 -rpcport=8332 -rpcuser=... -stdinrpcpass getblockhash 789070 exited with status 1 May 10 11:45:51 btc-2 systemd[1]: Stopping clightning.service... May 10 11:45:51 btc-2 systemd[1]: clightning.service: Deactivated successfully. May 10 11:45:51 btc-2 systemd[1]: Stopped clightning.service. May 10 11:45:51 btc-2 systemd[1]: clightning.service: Consumed 1min 52.006s CPU time, received 170.3M IP traffic, sent 17.0M IP traffic. May 10 11:46:05 btc-2 systemd[1]: Starting clightning.service... May 10 11:46:12 btc-2 lightningd[1557919]: INFO plugin-bcli: bitcoin-cli initialized and connected to bitcoind. May 10 11:46:13 btc-2 lightningd[1557919]: bitcoind has gone backwards from 789054 to 788984 blocks! May 10 11:46:13 btc-2 systemd[1]: clightning.service: Main process exited, code=exited, status=1/FAILURE May 10 11:47:36 btc-2 systemd[1]: clightning.service: start-post operation timed out. Terminating. May 10 11:47:36 btc-2 systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM May 10 11:47:36 btc-2 systemd[1]: clightning.service: Failed with result 'exit-code'. May 10 11:47:36 btc-2 systemd[1]: Failed to start clightning.service. May 10 11:47:36 btc-2 systemd[1]: clightning.service: Consumed 7.904s CPU time, received 2.8K IP traffic, sent 2.0K IP traffic. May 10 11:47:46 btc-2 systemd[1]: clightning.service: Scheduled restart job, restart counter is at 1. May 10 11:47:46 btc-2 systemd[1]: Stopped clightning.service. May 10 11:47:46 btc-2 systemd[1]: clightning.service: Consumed 7.904s CPU time, received 2.8K IP traffic, sent 2.0K IP traffic. May 10 11:47:46 btc-2 systemd[1]: Starting clightning.service... May 10 11:47:50 btc-2 lightningd[1558825]: INFO plugin-bcli: bitcoin-cli initialized and connected to bitcoind. May 10 11:47:54 btc-2 lightningd[1558825]: INFO plugin-monitor.py: Plugin monitor.py initialized May 10 11:47:54 btc-2 lightningd[1558825]: INFO plugin-cln-zapper: 1 May 10 11:47:54 btc-2 lightningd[1558825]: INFO plugin-cln-zapper: Waiting for index: 1 May 10 11:47:54 btc-2 systemd[1]: Started clightning.service. ``` `systemd` output: ``` ○ rtl.service Loaded: loaded (/etc/systemd/system/rtl.service; enabled; preset: enabled) Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago Duration: 1h 50min 17.037s Process: 1552092 ExecStartPre=/nix/store/ym21fvcrrga8026wf4d1kd34gfjjvqik-rtl-setup-config (code=exited, status=0/SUCCESS) Process: 1552096 ExecStart=/nix/store/f64civv948i5igz7hag91wm67pm7apfc-rtl-0.13.6/bin/rtl (code=killed, signal=TERM) Main PID: 1552096 (code=killed, signal=TERM) IP: 4.7M in, 76.4K out CPU: 2.015s May 10 08:38:50 btc-2 rtl[1552096]: [5/10/2023, 8:38:50 AM] INFO: Channels => Channels List Received. May 10 08:38:50 btc-2 rtl[1552096]: [5/10/2023, 8:38:50 AM] INFO: Peers => Peers with Alias Received. May 10 09:23:03 btc-2 rtl[1552096]: [5/10/2023, 9:23:03 AM] INFO: Channels => Getting Channel List Forwards... May 10 10:23:23 btc-2 rtl[1552096]: [5/10/2023, 10:23:23 AM] INFO: Authenticate => Logged out. May 10 10:24:23 btc-2 systemd[1]: Stopping rtl.service... May 10 10:24:23 btc-2 systemd[1]: rtl.service: Deactivated successfully. May 10 10:24:23 btc-2 systemd[1]: Stopped rtl.service. May 10 10:24:23 btc-2 systemd[1]: rtl.service: Consumed 2.015s CPU time, received 4.7M IP traffic, sent 76.4K IP traffic. May 10 10:26:13 btc-2 systemd[1]: Dependency failed for rtl.service. May 10 10:26:13 btc-2 systemd[1]: rtl.service: Job rtl.service/start failed with result 'dependency'. ○ spark-wallet.service Loaded: loaded (/etc/systemd/system/spark-wallet.service; enabled; preset: enabled) Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago Duration: 1h 50min 17.270s Process: 1552078 ExecStart=/nix/store/l4pcffnr3kgi4m084jm1sdkyhb8g9zyc-unit-script-spark-wallet-start/bin/spark-wallet-start (code=exited, status=0/SUCCESS) Main PID: 1552078 (code=exited, status=0/SUCCESS) IP: 2.9K in, 1.9K out CPU: 1.408s May 10 10:20:36 btc-2 spark-wallet-start[1552078]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) May 10 10:23:45 btc-2 spark-wallet-start[1552078]: Error: connect ETIMEDOUT 107.154.133.13:443 May 10 10:23:45 btc-2 spark-wallet-start[1552078]: at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1159:16) May 10 10:24:23 btc-2 systemd[1]: Stopping spark-wallet.service... May 10 10:24:23 btc-2 spark-wallet-start[1552078]: Caught SIGTERM, shutting down May 10 10:24:23 btc-2 systemd[1]: spark-wallet.service: Deactivated successfully. May 10 10:24:23 btc-2 systemd[1]: Stopped spark-wallet.service. May 10 10:24:23 btc-2 systemd[1]: spark-wallet.service: Consumed 1.408s CPU time, received 2.9K IP traffic, sent 1.9K IP traffic. May 10 10:26:13 btc-2 systemd[1]: Dependency failed for spark-wallet.service. May 10 10:26:13 btc-2 systemd[1]: spark-wallet.service: Job spark-wallet.service/start failed with result 'dependency'. ○ clightning-rest.service Loaded: loaded (/etc/systemd/system/clightning-rest.service; enabled; preset: enabled) Active: inactive (dead) since Wed 2023-05-10 10:24:23 UTC; 9h ago Duration: 1h 50min 17.148s Process: 1552077 ExecStartPre=/nix/store/hicrgp5c1ndbfwh2698bcwb5ny4k1h66-unit-script-clightning-rest-pre-start/bin/clightning-rest-pre-start (code=exited, status=0/SUCCESS) Process: 1552082 ExecStart=/nix/store/0rk0ldjxxf0xnb1pz3nfsh0ssghbg2jn-clightning-rest-0.9.0/bin/cl-rest (code=exited, status=0/SUCCESS) Process: 1552083 ExecStartPost=/nix/store/2pj40qdlkp7wj09apmlp5qqkr1dfak37-unit-script-clightning-rest-post-start/bin/clightning-rest-post-start (code=exited, status=0/SUCCESS) Main PID: 1552082 (code=exited, status=0/SUCCESS) IP: 19.2K in, 4.6M out CPU: 2.228s May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- Starting the cl-rest server --- May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- cl-rest api server is ready and listening on port: 3001 --- May 10 08:34:08 btc-2 clightning-rest[1552082]: warn: --- cl-rest doc server is ready and listening on port: 4001 --- May 10 10:24:23 btc-2 clightning-rest[1552082]: warn: Client disconnected, total clients: 0 May 10 10:24:23 btc-2 systemd[1]: Stopping clightning-rest.service... May 10 10:24:23 btc-2 systemd[1]: clightning-rest.service: Deactivated successfully. May 10 10:24:23 btc-2 systemd[1]: Stopped clightning-rest.service. May 10 10:24:23 btc-2 systemd[1]: clightning-rest.service: Consumed 2.228s CPU time, received 19.2K IP traffic, sent 4.6M IP traffic. May 10 10:26:13 btc-2 systemd[1]: Dependency failed for clightning-rest.service. May 10 10:26:13 btc-2 systemd[1]: clightning-rest.service: Job clightning-rest.service/start failed with result 'dependency'. ● bitcoind.service Loaded: loaded (/etc/systemd/system/bitcoind.service; enabled; preset: enabled) Active: active (running) since Wed 2023-05-10 11:46:05 UTC; 8h ago Process: 1557872 ExecStartPre=/nix/store/ns2xw9hd3j3qlv91qr4yzz057xc7j2id-unit-script-bitcoind-pre-start/bin/bitcoind-pre-start (code=exited, status=0/SUCCESS) Process: 1557910 ExecStartPost=/nix/store/s9j6ndhq1kpca6qyjgnssg2p12610wiv-unit-script-bitcoind-post-start/bin/bitcoind-post-start (code=exited, status=0/SUCCESS) Main PID: 1557879 (bitcoind) IP: 1.0G in, 38.2G out IO: 296.3G read, 1.0G written Tasks: 26 (limit: 9527) Memory: 6.1G CPU: 2h 13min 431ms CGroup: /system.slice/bitcoind.service └─1557879 /nix/store/a36chbn6hz23968jvh555c11xg28djy5-bitcoind-24.0.1/bin/bitcoind -datadir=/bitcoin May 10 19:37:20 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789133, peer=3711 (block-relay-only) May 10 19:46:56 btc-2 bitcoind[1557879]: Timeout downloading block 00000000000000000003bf94447ad5adc5859cd78d600000525851b7784a5599 from peer=3514, disconnecting May 10 19:47:03 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000003bf94447ad5adc5859cd78d600000525851b7784a5599 height=789133 version=0x3fff0000 log2_work=94.171474 tx=835591984 date='2023-05-10T19:36:48Z' progress=0.999998 cache=235.0MiB(1585031txo) May 10 19:48:42 btc-2 bitcoind[1557879]: UpdateTip: new best=000000000000000000002e547daae0c3421698dcdb55bc4af10debb08fd6530f height=789134 version=0x2d054000 log2_work=94.171488 tx=835595967 date='2023-05-10T19:48:28Z' progress=1.000000 cache=236.0MiB(1592518txo) May 10 19:51:14 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789134, peer=3824 (block-relay-only) May 10 19:52:57 btc-2 bitcoind[1557879]: ERROR: AcceptBlock: bad-witness-nonce-size, ContextualCheckBlock : invalid witness reserved value size May 10 19:52:57 btc-2 bitcoind[1557879]: ERROR: ProcessNewBlock: AcceptBlock FAILED (bad-witness-nonce-size, ContextualCheckBlock : invalid witness reserved value size) May 10 19:53:01 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000004a2b02d0d5d22631724ec0438118c54d10d8ba00124c5 height=789135 version=0x20a00000 log2_work=94.171501 tx=835599718 date='2023-05-10T19:52:32Z' progress=1.000000 cache=236.8MiB(1599138txo) May 10 19:53:59 btc-2 bitcoind[1557879]: UpdateTip: new best=00000000000000000003ec91f2476f9098129b1d9d68159842615807c4941baf height=789136 version=0x2001e000 log2_work=94.171514 tx=835603278 date='2023-05-10T19:53:45Z' progress=1.000000 cache=237.6MiB(1604592txo) May 10 19:54:34 btc-2 bitcoind[1557879]: New outbound peer connected: version: 70016, blocks=789136, peer=3847 (block-relay-only) ● clightning.service Loaded: loaded (/etc/systemd/system/clightning.service; enabled; preset: enabled) Active: active (running) since Wed 2023-05-10 11:47:54 UTC; 8h ago Process: 1558819 ExecStartPre=/nix/store/zm567pfnggl5nslxm1j0vls5636fzvn9-unit-script-clightning-pre-start/bin/clightning-pre-start (code=exited, status=0/SUCCESS) Process: 1558826 ExecStartPost=/nix/store/3lv8jihx427g5v4x2f5xzyl84jdx6khv-unit-script-clightning-post-start/bin/clightning-post-start (code=exited, status=0/SUCCESS) Main PID: 1558825 (lightningd) IP: 392.9M in, 73.9M out IO: 744.1M read, 793.5M written Tasks: 55 (limit: 9527) Memory: 465.4M CPU: 6min 9.548s CGroup: /system.slice/clightning.service ├─1558825 /nix/store/kw76ywkn387rz9izpnvcc7l5l4yyyrm3-clightning-23.02.2/bin/lightningd --lightning-dir=/var/lib/clightning ```