CLN crash loop after update-nix-bitcoin and deploy

[tompro]

After an update and deploy clightning is going into a crash loop waiting for the lightning-rpc file to appear. Logs do not show anything interesting at startup other than that Python is not present:

Nov 21 20:21:13 nixbtc systemd[1]: Starting clightning.service...
Nov 21 20:21:14 nixbtc lightningd[80922]: INFO    lightningd: v24.08.1
Nov 21 20:21:14 nixbtc lightningd[80922]: INFO    plugin-wss-proxy: Killing plugin: disabled itself: No python3 binary found
Nov 21 20:21:14 nixbtc lightningd[80922]: UNUSUAL plugin-bookkeeper: topic 'utxo_deposit' is not a known notification topic
Nov 21 20:21:14 nixbtc lightningd[80922]: UNUSUAL plugin-bookkeeper: topic 'utxo_spend' is not a known notification topic
Nov 21 20:21:25 nixbtc lightningd[80922]: INFO    plugin-bcli: bitcoin-cli initialized and connected to bitcoind.
Nov 21 20:22:44 nixbtc systemd[1]: clightning.service: start-post operation timed out. Terminating.
Nov 21 20:22:44 nixbtc systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
Nov 21 20:23:36 nixbtc systemd[1]: clightning.service: Failed with result 'timeout'.
Nov 21 20:23:36 nixbtc systemd[1]: Failed to start clightning.service.

[erikarvstedt]

I haven't see this bug in any of our nix-bitcoin systems.

Can you share your nix-bitcoin clightning config? Are you running nix-bitcoin on particularly slow hardware?

[tompro]

Thanks for the quick reply! I guess you mean the configuration.nix? It is essentially running with stock config and a bunch of connected services.

  nix-bitcoin.onionServices.clightning.public = true;
  services.clightning-rest = {
    enable = true;
    lndconnect = {
      enable = true;
      onion = true;
    };
    extraConfig = {
      DOMAIN = "******";
    };
    tor.enforce = false;
  };

  services.rtl = {
    enable = true;
    nodes.clightning.enable = true;
    nightTheme = true;
    address = "0.0.0.0";
    tor.enforce = false;
  };

  services.btcpayserver = {
    enable = true;
    lightningBackend = "clightning";
    tor.enforce = false;
   ...
  };

I am running this on a x86 desktop in a Proxmox vm with quite some resources assigned (4c, 16G). I do use a Sata SSD for the data though which is not at speed levels known from m2 SSDs.

Note: I tested skipping the socket loop by just creating a fake socket file myself, clightning starts (service is up) but all processes other than lightningd immediately become zombie processes. This is probably bcs of the fake socket file though.