Docs issues, "Manage Nethermind with systemd"

Working from https://docs.nethermind.io/nethermind/first-steps-with-nethermind/manage-nethermind-with-systemd, at v 1.13.6:

At the "Create a new user" stage, I happen to have a large SSD for chain data and a smaller one for the OS, as per common hardware recommendations for staking machines. Simply running sudo useradd -m -s /bin/bash nethermind will create the user's home dir as /home/nethermind which is on the smaller SSD with the OS. I used the -d switch to specify the home dir path. I suspect many users will be in the same boat.

At the "Install Dependencies" stage, I wasn't forced to set a password for user nethermind. When I try to sudo, I'm prompted for my password regardless. sudo can't work.

(Skipping this step because I've done it before creating the nethermind user anyway.)

At the "Get Nethermind" stage, I don't want to have to be continually checking for updates, downloading and installing them so I went with the Ubuntu PPA instead. Skipping this step.

At the "download the nethermind.service config" stage, the executable is /home/nethermind/build/Nethermind.Runner. But because I'm using the Ubuntu package, this should be /usr/bin/nethermind. I made this change to the supplied service file.

But I couldn't start the service:

$ sudo systemctl daemon-reload
$ sudo service nethermind start
$ journalctl -u nethermind -f
Aug 26 04:21:05 stake nethermind[18883]: sudo: a password is required
Aug 26 04:21:05 stake sudo[18883]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Aug 26 04:21:05 stake sudo[18883]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/share/nethermind/Nethermind.Runner --datadir /data/nethermind
Aug 26 04:21:05 stake systemd[1]: nethermind.service: Main process exited, code=exited, status=1/FAILURE
Aug 26 04:21:05 stake systemd[1]: nethermind.service: Failed with result 'exit-code'.
Aug 26 04:21:05 stake systemd[1]: nethermind.service: Scheduled restart job, restart counter is at 5.
Aug 26 04:21:05 stake systemd[1]: Stopped Nethermind Node.
Aug 26 04:21:05 stake systemd[1]: nethermind.service: Start request repeated too quickly.
Aug 26 04:21:05 stake systemd[1]: nethermind.service: Failed with result 'exit-code'.
Aug 26 04:21:05 stake systemd[1]: Failed to start Nethermind Node.

User nethermind still has no password and systemd seems to expect there to be one. So I set one for the user.

Then I realised I was in fact trying to start the service as my original user, not nethermind. So I su'ed to nethermind and tried again:

$ sudo service nethermind start
[sudo] password for nethermind: 
Sorry, user nethermind is not allowed to execute '/usr/sbin/service nethermind start' as root on stake.

Strange, because I did also add user nethermind to sudoers.

Giving up on running a systemd service at this point. Would love to get this working though. I want the process running if the machine reboots after a power cut.

@eliotstock Not sure why you're getting issues there.

All you need to do is to create a systemd unit like this in /etc/systemd/system/nethermind.service (example is for ropsten network):

[Unit]
Description=Nethermind Node
Documentation=https://docs.nethermind.io
After=network.target

[Service]
User=nethermind
Group=nethermind
EnvironmentFile=/home/nethermind/.env
WorkingDirectory=/home/nethermind
ExecStart=/home/nethermind/build/Nethermind.Runner --datadir /home/nethermind/.nethermind --config /home/nethermind/build/configs/ropsten.cfg --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true
Restart=on-failure
LimitNOFILE=1000000

[Install]
WantedBy=default.target

Also make sure to create /home/nethermind/.env with the following contents:

NETHERMIND_LOG_LEVEL=INFO
#NETHERMIND_JSONRPCCONFIG_ENABLEDMODULES=[Web3,Eth,Subscribe,Trace,Net]
NETHERMIND_METRICSCONFIG_ENABLED=true
NETHERMIND_METRICSCONFIG_NODENAME=Nethermind
NETHERMIND_METRICSCONFIG_PUSHGATEWAYURL=http://localhost:9090/metrics
NETHERMIND_HEALTHCHECKSCONFIG_ENABLED=false
NETHERMIND_PRUNINGCONFIG_CACHEMB=16384
NETHERMIND_ETHSTATSCONFIG_ENABLED=false
NETHERMIND_ETHSTATSCONFIG_SERVER=http://localhost:3000/api

As for sudoers, in order to run from nethermind user you need to have the following in /etc/sudoers.d/nethermind:

Cmnd_Alias RESTART = /bin/systemctl restart nethermind,/bin/systemctl start nethermind,/bin/systemctl stop nethermind
nethermind  ALL=(root)  NOPASSWD: RESTART

Make sure to also:

chmod 0440 /etc/sudoers.d/nethermind

Otherwise sudo won't work.

Then, from nethermind user it works just fine:

sudo systemctl start nethermind
sudo systemctl stop nethermind
sudo systemctl restart nethermind

Thanks, @gituser, but I followed this more or less exactly, changing only the paths for my machine, and still wasn't able to start the service:

$ journalctl -u nethermind -f
Sep 03 00:08:28 stake nethermind[2010]: sudo: a password is required
Sep 03 00:08:28 stake sudo[2010]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Sep 03 00:08:28 stake sudo[2010]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/share/nethermind/Nethermind.Runner --datadir /data/nethermind --config /usr/share/nethermind/configs/ropsten.cfg --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true
Sep 03 00:08:28 stake systemd[1]: nethermind.service: Main process exited, code=exited, status=1/FAILURE
Sep 03 00:08:28 stake systemd[1]: nethermind.service: Failed with result 'exit-code'.
Sep 03 00:08:28 stake systemd[1]: nethermind.service: Scheduled restart job, restart counter is at 5.
Sep 03 00:08:28 stake systemd[1]: Stopped Nethermind Node.
Sep 03 00:08:28 stake systemd[1]: nethermind.service: Start request repeated too quickly.
Sep 03 00:08:28 stake systemd[1]: nethermind.service: Failed with result 'exit-code'.
Sep 03 00:08:28 stake systemd[1]: Failed to start Nethermind Node.

I wondered if running that COMMAND from the logs in a shall as root would work. It did:

$ sudo -i
root@stake:~# 
root@stake:~# cd /data/nethermind/
root@stake:/data/nethermind# /usr/share/nethermind/Nethermind.Runner --datadir /data/nethermind --config /usr/share/nethermind/configs/ropsten.cfg --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true
2022-09-03 00-05-02.2891|Nethermind starting initialization.
2022-09-03 00-05-02.3367|Loading embedded plugins
...

So it runs on the shell as root using a direct command line, but not as nethermind when using sudo systemctl start nethermind. /etc/sudoers.d/nethermind is definitely mode 0440.

@eliotstock

did you paste into /etc/sudoers.d/nethermind the content from my post?

try deleting and re-adding nethermind user:

userdel nethermind
adduser --disabled-password nethermind

@gituser, yes, /etc/sudoers.d/nethermind has your change:

$ sudo more /etc/sudoers.d/nethermind 
Cmnd_Alias RESTART = /bin/systemctl restart nethermind,/bin/systemctl start nethermind,/bin/systemctl sto
p nethermind
nethermind  ALL=(root)  NOPASSWD: RESTART

I tried deleting and re-adding the nethermind user and having it have no password. Starting the service as the user fails:

$ sudo systemctl start nethermind
$ journalctl -u nethermind -f
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
Sep 04 00:39:42 stake sudo[3362]: pam_unix(sudo:auth): conversation failed
Sep 04 00:39:42 stake nethermind[3362]: sudo: a password is required
Sep 04 00:39:42 stake sudo[3362]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Sep 04 00:39:42 stake sudo[3362]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/bin/chown -R nethermind /usr/share/nethermind
Sep 04 00:39:42 stake nethermind[3360]: Executing Nethermind Runner
Sep 04 00:39:42 stake nethermind[3363]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
Sep 04 00:39:42 stake sudo[3363]: pam_unix(sudo:auth): conversation failed
Sep 04 00:39:42 stake nethermind[3363]: sudo: a password is required
Sep 04 00:39:42 stake sudo[3363]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Sep 04 00:39:42 stake sudo[3363]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/share/nethermind/Nethermind.Runner --datadir /data/nethermind --config /usr/share/nethermind/configs/ropsten.cfg --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true

Seeing sudo: a password is required, I also tried this with the nethermind user having a password:

sudo -i
# 
# passwd nethermind
New password: 
Retype new password: 
passwd: password updated successfully

...end the interactive session as root and become user nethermind again...

$ sudo systemctl start nethermind
nethermind@stake:~$ journalctl -u nethermind -f
Hint: You are currently not seeing messages from other users and the system.
      Users in groups 'adm', 'systemd-journal' can see all messages.
      Pass -q to turn off this notice.
Sep 04 00:45:47 stake sudo[3438]: pam_unix(sudo:auth): conversation failed
Sep 04 00:45:47 stake nethermind[3438]: sudo: a password is required
Sep 04 00:45:47 stake sudo[3438]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Sep 04 00:45:47 stake sudo[3438]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/bin/chown -R nethermind /usr/share/nethermind
Sep 04 00:45:47 stake nethermind[3436]: Executing Nethermind Runner
Sep 04 00:45:47 stake nethermind[3439]: sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
Sep 04 00:45:47 stake sudo[3439]: pam_unix(sudo:auth): conversation failed
Sep 04 00:45:47 stake nethermind[3439]: sudo: a password is required
Sep 04 00:45:47 stake sudo[3439]: pam_unix(sudo:auth): auth could not identify password for [nethermind]
Sep 04 00:45:47 stake sudo[3439]: nethermind : command not allowed ; PWD=/data/nethermind ; USER=root ; COMMAND=/usr/share/nethermind/Nethermind.Runner --datadir /data/nethermind --config /usr/share/nethermind/configs/ropsten.cfg --JsonRpc.Enabled true --HealthChecks.Enabled true --HealthChecks.UIEnabled true

So the problem is the same whether the nethermind user has a password or not: sudo: a password is required.

I also then noticed that the path to systemctl in your /etc/sudoers.d/nethermind is wrong for my host. Mine is at /usr/bin/systemctl, not /bin/systemctl. So I fixed that and ran sudo visudo -c again and tested again. Still no joy.

Of course there's also the issue that the process is trying to run /usr/bin/chown -R nethermind /usr/share/nethermind and this command is not in the sudoers file yet. That's also unesseary in my case - it's already owned by nethermind.

Could it be that no-one is really running this using systemd, or if they are they're just running it as root for now?

same issues here: client doesn't start using ubuntu PPA guide.

@gituser either documentation must be updated to give predictable working result or client bin's behaviour under the hood.

Added as a urgent item to our Documentation project - we will rework this entire section and merge it with "Validator setup -> Eth2". You can check if this would address Your problems: https://docs.nethermind.io/nethermind/guides-and-helpers/validator-setup/eth2-validator

@shashankshampi Please go through this docs, reproduce it and see if we can make it more user friendly

NethermindEth / nethermind

Docs issues, "Manage Nethermind with systemd" #4482