Duplicate Lighthouse service startup parameters

[m1is]

There seems to be an issue after running the reset-monitoring.sh script. The script itself runs fine however when I rerun the monitoring setup script the Lighthouse Validator/Beacon services fail to start. There seems to be a problem with duplicate startup parameters, I'm seeing the following in the logs:

Aug 08 11:35:03 pls-node-1 systemd[1]: lighthouse-beacon.service: Failed with result 'exit-code'. Aug 08 11:35:08 pls-node-1 systemd[1]: lighthouse-beacon.service: Scheduled restart job, restart counter is at 24. Aug 08 11:35:08 pls-node-1 systemd[1]: Stopped Lighthouse Beacon. Aug 08 11:35:08 pls-node-1 systemd[1]: Started Lighthouse Beacon. Aug 08 11:35:08 pls-node-1 lh[69486]: error: The argument '--validator-monitor-auto' was provided more than once, but cannot be used multiple times

Aug 08 11:35:35 pls-node-1 systemd[1]: lighthouse-validator.service: Failed with result 'exit-code'. Aug 08 11:35:40 pls-node-1 systemd[1]: lighthouse-validator.service: Scheduled restart job, restart counter is at 30. Aug 08 11:35:40 pls-node-1 systemd[1]: Stopped Lighthouse Validator. Aug 08 11:35:40 pls-node-1 systemd[1]: Started Lighthouse Validator. Aug 08 11:35:40 pls-node-1 lh[69512]: error: The argument '--metrics' was provided more than once, but cannot be used multiple times

[rhmaxdotorg]

@m1is hey there, thanks for reporting this!

Apologies for this error. I haven't came across it on my end, so I'm not sure what's causing it.

Can you paste the original config that is giving you the errors so I can see what is getting duplicated?

How I think you can fix it

I'm kinda working blind here as it's not in front of me, but here goes...

At first glance, it sounds like there are duplicate parameters in the services files for Lighthouse. Good news this is an easy fix, but you'll need to remove them manually.

Step 1: edit the service files, one at a time

$ sudo pico /etc/systemd/system/lighthouse-beacon.service

Check out Line 12 with ExecStart= and look at the end of the line to find any duplicate parameters. For example, if you see --validator-monitor-auto twice or more, make sure there's only one of them.

Then once you're finished making changes, do ctrl+x and then 'y' to exit and save the changes.

$ sudo pico /etc/systemd/system/lighthouse-validator.service

Same thing, check the ExecStart= line for duplicate parameters and make sure there is only one of them. In this case, the logs are saying there is more than one --metrics parameter.

Then you can restart both lighthouse services.

sudo systemctl restart lighthouse-beacon lighthouse-validator

And make sure you check the logs again to make sure they have been successfully restarted and working.

Let me know how it goes! Hope to get your validator back up and running ASAP.

FWIW the script itself makes a backup of each service file when you run the monitoring installation script and the reset script simply restores the original setup, so I'm not sure how that went wrong.

# restore service backup files and restart client services
sudo mv /etc/systemd/system/geth.service.BACKUP /etc/systemd/system/geth.service
sudo mv /etc/systemd/system/lighthouse-beacon.service.BACKUP /etc/systemd/system/lighthouse-beacon.service
sudo mv /etc/systemd/system/lighthouse-validator.service.BACKUP /etc/systemd/system/lighthouse-validator.service

Also, is Geth is working fine and no duplicate errors for it?

[m1is]

Looks like the duplicate parameters got appended to the end of the start up line like you said, I removed them and now everything is up and running. What threw me off was when I ran the monitoring setup but didn't set it up for port forwarding to the outside world. I was trying to access Grafana on port 8080 not realizing I should be using 3000 internally. So I got frustrated and ran the reset monitoring script, then ran the setup again thinking it might fix something only to realize my mistake. Not sure if those would be the exact steps to reproduce the issue but might be a good place to try to troubleshoot how it might happen. Anyway, good to know a little more about how to tinker with the service files and troubleshoot issues. Thanks for your help.

[rhmaxdotorg]

Glad you're back up and running!

So if I got this right, these were the steps: 1) run monitoring setup script 2) run reset monitoring script 3) run monitoring setup script again

So the monitoring setup script backs up the original service files like this (so later they can be restored with the paired reset script):

# backup service files, update /w metrics flag and reload services
sudo cp /etc/systemd/system/geth.service /etc/systemd/system/geth.service.BACKUP
sudo cp /etc/systemd/system/lighthouse-beacon.service /etc/systemd/system/lighthouse-beacon.service.BACKUP
sudo cp /etc/systemd/system/lighthouse-validator.service /etc/systemd/system/lighthouse-validator.service.BACKUP

And then reset script does this:

# restore service backup files and restart client services
sudo mv /etc/systemd/system/geth.service.BACKUP /etc/systemd/system/geth.service
sudo mv /etc/systemd/system/lighthouse-beacon.service.BACKUP /etc/systemd/system/lighthouse-beacon.service
sudo mv /etc/systemd/system/lighthouse-validator.service.BACKUP /etc/systemd/system/lighthouse-validator.service

So running the monitoring setup script again would just copy the (restored and original) service files to the .BACKUP files again and then modify the services files to add the parameters. I still don't see how duplicate parameters could exist at that point, hmm.

Anyways, will close this for now and if I think come across this happening again or see something I'm missing, update the scripts. Thanks again for letting me know and hope you got Grafana setup and accessible now!