search

ElementsProject / lightning

Core Lightning — Lightning Network implementation focusing on spec compliance and performance
Other
2.85k stars 901 forks source link

(raspiblitz upgrade?) lightningd does not start anymore #6324

Closed orangedmind closed 1 year ago

orangedmind commented 1 year ago

I updated to CLN v23.02.2. Since this update I cannot start lightningd anymore:

image

I already reinstalled CLN from scratch and used the cl rescue file to restore CLN wallet but no success.

Any ideas?

wtogami commented 1 year ago

Please find your CLN log file. If you didn't configure one it might instead be logging to journal. There are likely error messages in there to tell you what's going on. Don't post the entire log here because it might have private details.

orangedmind commented 1 year ago

cl.log does not contain any entries since the day I made the update.

image

I reinstalled today and used the cl rescue file. Below you can find the logs of the install process:

Provisioning Core Lightning Binary - run config script

Running: 'cl.install.sh install'

INSTALL CORE LIGHTNING v23.02.2 BINARY

only binary install to system

no configuration, no systemd service

Core Lightning binary already installed - done Provisioning CL Mainnet - run config script

Running: 'cl.install.sh on mainnet'

Using the settings for: bitcoin mainnet

Running: 'cl.install.sh install'

INSTALL CORE LIGHTNING v23.02.2 BINARY

only binary install to system

no configuration, no systemd service

Core Lightning binary already installed - done

Make sure bitcoin is in the debian-tor group

Add plugin-dir: /home/bitcoin/cl-plugins-enabled

Add plugin-dir: /home/bitcoin/cl-plugins-available

Store the lightning data in /mnt/hdd/app-data/.lightning

Symlink to /home/bitcoin/

/home/bitcoin/.lightning/config

The file /home/bitcoin/.lightning/config is already present

/home/bitcoin/.lightning/bitcoin/hsm_secret

/home/bitcoin/.lightning/bitcoin/hsm_secret is already present

hint: Pulling without specifying how to reconcile divergent branches is hint: discouraged. You can squelch this message by running one of the following hint: commands sometime before your next pull: hint: hint: git config pull.rebase false # merge (the default strategy) hint: git config pull.rebase true # rebase hint: git config pull.ff only # fast-forward only hint: hint: You can replace "git config" with "git config --global" to set a default hint: preference for all repositories. You can also pass --rebase, --no-rebase, hint: or --ff-only on the command line to override the configured default per hint: invocation. Updating 3000327..1713f9a Fast-forward .github/workflows/main.yml | 8 +-- .mergify.yml | 12 ++--- README.md | 4 ++ backup/poetry.lock | 14 ++--- clearnet/README.md | 10 ++++ clearnet/clearnet.py | 114 ++++++++++++++++++++++++++++++++++++++++ clearnet/requirements.txt | 1 + clearnet/test_clearnet.py | 22 ++++++++ currencyrate/README.md | 4 +- currencyrate/currencyrate.py | 6 --- donations/poetry.lock | 12 +++-- feeadjuster/feeadjuster.py | 100 ++++++++++++++++++++--------------- feeadjuster/test_feeadjuster.py | 69 +++++++++++++++++++----- historian/poetry.lock | 12 +++-- noise/README.org | 14 +++-- noise/noise.py | 13 +++++ noise/requirements-dev.txt | 2 + noise/requirements.txt | 1 + noise/test_chat.py | 4 +- nostrify | 2 +- summary/summary.py | 39 +++++++++----- summary/test_summary.py | 22 ++++++++ 22 files changed, 373 insertions(+), 112 deletions(-) create mode 100644 clearnet/README.md create mode 100755 clearnet/clearnet.py create mode 100644 clearnet/requirements.txt create mode 100644 clearnet/test_clearnet.py create mode 100644 noise/requirements-dev.txt create mode 100644 noise/requirements.txt HEAD is now at 3000327 summary: check and dismiss old datastores

Checking dependencies

Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already satisfied: pip in /usr/local/lib/python3.9/dist-packages (23.1.2) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Defaulting to user installation because normal site-packages is not writeable Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already satisfied: pyln-client in /home/bitcoin/.local/lib/python3.9/site-packages (23.5) Requirement already satisfied: tqdm in /home/bitcoin/.local/lib/python3.9/site-packages (4.65.0) Requirement already satisfied: pyln-bolt7>=1.0 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-client) (1.0.246) Requirement already satisfied: pyln-proto>=0.12 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-client) (23.5) Requirement already satisfied: PySocks<2.0.0,>=1.7.1 in /usr/local/lib/python3.9/dist-packages (from pyln-proto>=0.12->pyln-client) (1.7.1) Requirement already satisfied: base58<3.0.0,>=2.1.1 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-proto>=0.12->pyln-client) (2.1.1) Requirement already satisfied: bitstring<4.0.0,>=3.1.9 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-proto>=0.12->pyln-client) (3.1.9) Requirement already satisfied: coincurve<18.0.0,>=17.0.0 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-proto>=0.12->pyln-client) (17.0.0) Requirement already satisfied: cryptography<37.0.0,>=36.0.1 in /home/bitcoin/.local/lib/python3.9/site-packages (from pyln-proto>=0.12->pyln-client) (36.0.2) Requirement already satisfied: asn1crypto in /home/bitcoin/.local/lib/python3.9/site-packages (from coincurve<18.0.0,>=17.0.0->pyln-proto>=0.12->pyln-client) (1.5.1) Requirement already satisfied: cffi>=1.3.0 in /home/bitcoin/.local/lib/python3.9/site-packages (from coincurve<18.0.0,>=17.0.0->pyln-proto>=0.12->pyln-client) (1.15.1) Requirement already satisfied: pycparser in /home/bitcoin/.local/lib/python3.9/site-packages (from cffi>=1.3.0->coincurve<18.0.0,>=17.0.0->pyln-proto>=0.12->pyln-client) (2.21) Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Requirement already satisfied: poetry in /usr/local/lib/python3.9/dist-packages (1.4.2) Requirement already satisfied: build<0.11.0,>=0.10.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.10.0) Requirement already satisfied: cachecontrol[filecache]<0.13.0,>=0.12.9 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.12.11) Requirement already satisfied: cleo<3.0.0,>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (2.0.1) Requirement already satisfied: crashtest<0.5.0,>=0.4.1 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.4.1) Requirement already satisfied: dulwich<0.22.0,>=0.21.2 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.21.5) Requirement already satisfied: filelock<4.0.0,>=3.8.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (3.12.0) Requirement already satisfied: html5lib<2.0,>=1.0 in /usr/lib/python3/dist-packages (from poetry) (1.1) Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.9/dist-packages (from poetry) (6.6.0) Requirement already satisfied: installer<0.8.0,>=0.7.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.7.0) Requirement already satisfied: jsonschema<5.0.0,>=4.10.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (4.17.3) Requirement already satisfied: keyring<24.0.0,>=23.9.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (23.13.1) Requirement already satisfied: lockfile<0.13.0,>=0.12.2 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.12.2) Requirement already satisfied: packaging>=20.4 in /usr/local/lib/python3.9/dist-packages (from poetry) (23.1) Requirement already satisfied: pexpect<5.0.0,>=4.7.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (4.8.0) Requirement already satisfied: pkginfo<2.0.0,>=1.9.4 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.9.6) Requirement already satisfied: platformdirs<3.0.0,>=2.5.2 in /usr/local/lib/python3.9/dist-packages (from poetry) (2.6.2) Requirement already satisfied: poetry-core==1.5.2 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.5.2) Requirement already satisfied: poetry-plugin-export<2.0.0,>=1.3.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.3.1) Requirement already satisfied: pyproject-hooks<2.0.0,>=1.0.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.0.0) Requirement already satisfied: requests<3.0,>=2.18 in /usr/local/lib/python3.9/dist-packages (from poetry) (2.30.0) Requirement already satisfied: requests-toolbelt<0.11.0,>=0.9.1 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.10.1) Requirement already satisfied: shellingham<2.0,>=1.5 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.5.0.post1) Requirement already satisfied: tomli<3.0.0,>=2.0.1 in /usr/local/lib/python3.9/dist-packages (from poetry) (2.0.1) Requirement already satisfied: tomlkit!=0.11.2,!=0.11.3,<1.0.0,>=0.11.1 in /usr/local/lib/python3.9/dist-packages (from poetry) (0.11.8) Requirement already satisfied: trove-classifiers>=2022.5.19 in /usr/local/lib/python3.9/dist-packages (from poetry) (2023.5.2) Requirement already satisfied: urllib3<2.0.0,>=1.26.0 in /usr/local/lib/python3.9/dist-packages (from poetry) (1.26.15) Requirement already satisfied: virtualenv!=20.4.5,!=20.4.6,<21.0.0,>=20.4.3 in /usr/local/lib/python3.9/dist-packages (from poetry) (20.21.1) Requirement already satisfied: msgpack>=0.5.2 in /usr/local/lib/python3.9/dist-packages (from cachecontrol[filecache]<0.13.0,>=0.12.9->poetry) (1.0.5) Requirement already satisfied: rapidfuzz<3.0.0,>=2.2.0 in /usr/local/lib/python3.9/dist-packages (from cleo<3.0.0,>=2.0.0->poetry) (2.15.1) Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.9/dist-packages (from importlib-metadata>=4.4->poetry) (3.15.0) Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5.0.0,>=4.10.0->poetry) (23.1.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.9/dist-packages (from jsonschema<5.0.0,>=4.10.0->poetry) (0.19.3) Requirement already satisfied: jaraco.classes in /usr/local/lib/python3.9/dist-packages (from keyring<24.0.0,>=23.9.0->poetry) (3.2.3) Requirement already satisfied: SecretStorage>=3.2 in /usr/local/lib/python3.9/dist-packages (from keyring<24.0.0,>=23.9.0->poetry) (3.3.3) Requirement already satisfied: jeepney>=0.4.2 in /usr/local/lib/python3.9/dist-packages (from keyring<24.0.0,>=23.9.0->poetry) (0.8.0) Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.9/dist-packages (from pexpect<5.0.0,>=4.7.0->poetry) (0.7.0) Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.9/dist-packages (from requests<3.0,>=2.18->poetry) (3.1.0) Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests<3.0,>=2.18->poetry) (2.8) Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests<3.0,>=2.18->poetry) (2020.6.20) Requirement already satisfied: distlib<1,>=0.3.6 in /usr/local/lib/python3.9/dist-packages (from virtualenv!=20.4.5,!=20.4.6,<21.0.0,>=20.4.3->poetry) (0.3.6) Requirement already satisfied: cryptography>=2.0 in /usr/lib/python3/dist-packages (from SecretStorage>=3.2->keyring<24.0.0,>=23.9.0->poetry) (3.3.2) Requirement already satisfied: more-itertools in /usr/local/lib/python3.9/dist-packages (from jaraco.classes->keyring<24.0.0,>=23.9.0->poetry) (9.1.0) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv Installing dependencies from lock file Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run poetry lock [--no-update] to fix it.

No dependencies to install or update

Stop the lightningd.service

Backup the existing old backup on the SDcard

/home/bitcoin/.lightning/bitcoin/backup.lock

Initialize the backup plugin

Found an existing database at /home/bitcoin/.lightning/bitcoin/lightningd.sqlite3, initializing the backup with a snapshot Successfully written initial snapshot to file:///home/bitcoin/lightningd.sqlite3.backup Initialized backup backend file:///home/bitcoin/lightningd.sqlite3.backup, you can now start Core-Lightning

The crontab for admin now is:

@weekly /home/admin/config.scripts/cl-plugin.backup.sh backup-compact mainnet

Removed /etc/systemd/system/multi-user.target.wants/lightningd.service.

Create /etc/systemd/system/lightningd.service

[Unit] Description=c-lightning daemon on mainnet Requires=bitcoind.service After=bitcoind.service Wants=network-online.target After=network-online.target

[Service] ExecStartPre=-/home/admin/config.scripts/cl.check.sh prestart mainnet ExecStart=/bin/sh -c '/usr/local/bin/lightningd --conf=/home/bitcoin/.lightning/config --pid-file=/run/lightningd/lightningd.pid --rpc-file-mode 0660' ExecStartPost=-/home/admin/config.scripts/cl.check.sh poststart mainnet

Creates /run/lightningd owned by bitcoin

RuntimeDirectory=lightningd

User=bitcoin Group=bitcoin

Type=forking hangs on restart

Type=simple PIDFile=/run/lightningd/lightningd.pid Restart=on-failure

TimeoutSec=240 RestartSec=30 StandardOutput=null StandardError=journal

Hardening measures

####################

Provide a private /tmp and /var/tmp.

PrivateTmp=true

Mount /usr, /boot/ and /etc read-only for the process.

ProtectSystem=full

Disallow the process and all of its children to gain

new privileges through execve().

NoNewPrivileges=true

Use a new /dev namespace only populated with API pseudo devices

such as /dev/null, /dev/zero and /dev/random.

PrivateDevices=true

Deny the creation of writable and executable memory mappings.

MemoryDenyWriteExecute=true

[Install] WantedBy=multi-user.target

Created symlink /etc/systemd/system/multi-user.target.wants/lightningd.service → /etc/systemd/system/lightningd.service.

Enabled the lightningd.service

Set logrotate for lightningd

cl.log.1 cl.log.2 cl.log.3 cl.log.4 cl.log.5 /home/bitcoin/.lightning/bitcoin/cl.log { rotate 5 daily copytruncate missingok olddir /home/bitcoin/.lightning/bitcoin/cl.log_old notifempty nocompress sharedscripts su bitcoin bitcoin }

The installed Core Lightning version is: v23.02.2

To activate the aliases reopen the terminal or use:

source ~/_aliases

Monitor the lightningd with:

sudo journalctl -fu lightningd sudo systemctl status lightningd

logs:

sudo tail -f /home/bitcoin/.lightning/bitcoin/cl.log

for the command line options use

lightning-cli help

wtogami commented 1 year ago

I had never before seen cl-install.sh so I did a quick search.

image

You appear to have previously installed CLN via Raspiblitz and are trying to upgrade using their scripts again?

You filed a ticket against the upstream lightningd project. This project is not responsible for how distributors package CLN. You need to ask for help from your distributor. Nobody here can help you.

openoms commented 1 year ago

You will see why lightningd is not starting in the journal:

sudo journalctl -fu lightningd -n 30

The raspiblitz has plenty of debug tools. Open an issue in the raspiblitz repo of need further help.