Startup order of systemd services need to be controlled

[ronnylov]

Testing creating exit node by using debian packages. I noticed that by default there is no control of what order the services are started. This made lethean-vpn-server.service to fail at reboot because it had no connection to wallet and reason for that was that lethean-wallet-vpn-rpc.server was not yet finnished staring up. Restarting lethean-vpn-server manually worked because then lethean-wallet-vpn-rpc was then finnished.

lethean-wallet-vpn-rpc.service should wait for network connection is available when using remote sync node and it does not harm waiting for it even when using local daemon. If it use local daemon then it should wait for lethean-daemon.service. Since we don't know if user choose to use lethean-daemon we can't add that to the systemd unit file but maybe make sure it have long enough timeout before it gives up? We may have to look more at how to do it in a reliable way. Alternatively instruct users to add dependency the lethean-daemon.service (which is a better solution to make sure it work but risk is people don't do it becuae they don't read instructions).

Anyway this is how I customize lethean-wallet-vpn-rpc.service when using remote sync node:

[Unit]
Description=LTHN wallet VPN RPC daemon
Wants=network-online.target
After=syslog.target network.target network-online.target lethean-daemon.service

[Service]
EnvironmentFile=/etc/default/lethean-wallet-vpn-rpc
Type=simple
WorkingDirectory=/var/lib/lthn
ExecStartPre=/bin/rm -f /var/lib/lthn/lethean-wallet-vpn-rpc.*.login
ExecStart=/usr/bin/lethean-wallet-vpn-rpc --vpn-rpc-bind-port ${RPCPORT} --password ${WALLETPASS} --rpc-login ${RPCLOGIN} --daemon-host ${DAEMONHOST} --wallet-file ${LETHEANWALLET} --log-file /var/log/lthn/wallet-vpn-rpc.log

User=lthn
Group=lthn
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog

[Install]
WantedBy=multi-user.target

The Wants and After lines have been changed to make sure network is online before it is started. If using lethean-daemon.service then this service should also be added to Wants and After lines. We should also setup lethean-daemon.service to wait for network-online.target in similar way.

Finally we set up lethean-vpn-server.service to depend on lethean-wallet-vpn-rpc.service and make sure it starts after lethean-wallet-vpn-rpc.service.

[Unit]
Description=LTHN VPN server
ConditionPathExists=/usr/bin/lthnvpnd
Requires=lethean-wallet-vpn-rpc.service
After=syslog.target lethean-wallet-vpn-rpc.service

[Service]
EnvironmentFile=/etc/default/lethean-vpn-server
Type=simple
ExecStartPre=/bin/sleep 10
ExecStart=/usr/bin/lthnvpnd $LTHNVPND_ARGS
ExecStopPost=/bin/rm -f /var/run/lthn/lthnvpnd.pid
User=lthn
Restart=always
RestartSec=20
StandardOutput=syslog
StandardError=syslog
PIDFile=/var/run/lthn/lthnvpnd.pid

[Install]
WantedBy=multi-user.target

lethean-wallet-vpn-rpc.service was added to Wants and After to make sure lethean-vpn-server.service starts afterwards.

EDIT changed Wants to Requires. I still had problems after doing this. It seems lethean-vpn-server.service starts directly after lethean-wallet-vpn-rpc.service just have begun to start and lethean-vpn-server tries to restart a few times but gives up before RPC service have completed the setup.

So I added a line in lethean-vpn-server.service: RestartSec=20 We could also add a delay in ExecStartPre or perhaps doping both: ExecStartPre=/bin/sleep 20

Hmm doing both maybe give double delay time? Decided to set ExecStartPre to 10 seconds and RestartSec to 20. If it get 20 or 30 seconds between retrials don't matter much. Also added 10 seconds Restartsec to lethean-wallet-vpn-rpc.service

EDIT: I could add lethean-daemon.service to After line in lethean-wallet-vpn-rpc.service as long as I did not add it to Wants line and still use it without running lethean-daemon.service. Should help with start order if you run daemon but still allow usage without daemon.

[ronnylov]

This article is interesting: https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6

To avoid it giving up restarting after a few trals it should not try to restart more than 5 times in any 10 second period (with default settings StartLimitBurst=5 StartLimitIntervalSec=10). If we set RestartSec values of 3 seconds or more it will never give up.Always learn something new :-)

[limosek]

Things are not so easy here. There can be lot of combinations. Like use external daemon (so dependency to daemon is not needed) or use external wallet URI (so dependency to wallet-rpc ins not needed)

You already mentioned. It will need some more testing here. Can you create pull request with finalised and tested unit files please? But it should be dependant only in case that external services are not used.

[ronnylov]

I will test new packages. But my suggested unit files worked for me with or without external daemon. The after line is relevant only for enabled services, it does not enable or start them.

[ronnylov]

I wait for the fix of deploy script and will test next revision.

[ronnylov]

Not yet fixed in debian packages lethean-vpn version 3.1.0-8 and lethean-wallet-vpn-rpc version 3.1.0-3.

Following files was modified in my example above: /lib/systemd/system/lethean-vpn-server.service /lib/systemd/system/lethean-wallet-vpn-rpc.service

The lethean-wallet-vpn-rpc.service is normally used no matter if local daemon or remote daemon. I think most important change is adding the RestartSec=20 line to lethean-vpn-server.service to avoid that it gives up trying before the wallet is ready. The other changes I suggested is more to make it succeed on first try. The problem I had was that lethean-vpn-server retries very quickly and gives up before lethean-wallet-vpn-rpc completes first run.

[ronnylov]

Tested debian package lethean-vpn version 3.1.0-14 and lethean-wallet-vpn-rpc version 3.1.0-4. There was no problems with start of services after reboot. These issues are solved as far as I can see.

[ronnylov]

Closing this issue.