[FR] [CAD-2490] - Multiple node processes on the same server

[johnalotoski]

Internal

Area Other (Configuration)

Describe the feature you'd like

• Node to support running multiple node processes on the same server, perhaps utilizing the same socket
• This would enable larger relay servers to better utilize available physical cores and compute
• This would then reduce the amount of horizontal server scaling required to support a large number of relays

[newhoggy]

What are the current obstacles to running multiple node processes on the same server? The chairman tests currently run multiple node processes on the same server, although they use different sockets.

[rdlrt]

@johnalotoski - You can run multiple nodes on same server, its just not recommended (for pool operators specifically as atleast some instance seen where 5 pools are being run on same machine - which becomes a decentralisation issue - as a workaround to nOpt ). Not sure why you'd want to share same socket tho.

[newhoggy]

JIRA ticket https://jira.iohk.io/browse/CAD-2490

[newhoggy]

Some resources for my own records:

• https://cardano-community.github.io/guild-operators/#/
• https://github.com/systemd/systemd/issues/8096
• https://gist.github.com/gartnera/31cafc3d0dcdfb5bf0f5a2192e3fbe2c
• https://unrouted.io/ops/2017/04/26/multicore-twistd-with-systemd/

[newhoggy]

Confirmation of this working

Pre-requisites

• Install cardano-node with systemd per: https://cardano-community.github.io/guild-operators/#/basics?id=pre-requisites https://cardano-community.github.io/guild-operators/#/Build/node-cli

This has been tested with a version of cardano-node later than 1.25.1, although cardano-node-1.25.1 should work. If not, try master.

Steps

Copy /opt/cardano/cnode/files/config.json to /opt/cardano/cnode/files/config-1.json, /opt/cardano/cnode/files/config-2.json and /opt/cardano/cnode/files/config-3.json, with the following respective changes:

$ diff config.json config-1.json 
23a24
>   "TraceDiffusionInitialization": true,
50c51
<       "/opt/cardano/cnode/logs/node0.json"
---
>       "/opt/cardano/cnode/logs/node-1.json"
53c54
<   "hasEKG": 12788,
---
>   "hasEKG": 12781,
56c57
<     12798
---
>     12791
86c87
<       "scName": "/opt/cardano/cnode/logs/node0.json",
---
>       "scName": "/opt/cardano/cnode/logs/node-1.json",

$ diff config.json config-2.json 
23a24
>   "TraceDiffusionInitialization": true,
50c51
<       "/opt/cardano/cnode/logs/node0.json"
---
>       "/opt/cardano/cnode/logs/node-2.json"
53c54
<   "hasEKG": 12788,
---
>   "hasEKG": 12782,
56c57
<     12798
---
>     12792
86c87
<       "scName": "/opt/cardano/cnode/logs/node0.json",
---
>       "scName": "/opt/cardano/cnode/logs/node-2.json",

$ diff config.json config-3.json 
23a24
>   "TraceDiffusionInitialization": true,
50c51
<       "/opt/cardano/cnode/logs/node0.json"
---
>       "/opt/cardano/cnode/logs/node-3.json"
53c54
<   "hasEKG": 12788,
---
>   "hasEKG": 12783,
56c57
<     12798
---
>     12793
86c87
<       "scName": "/opt/cardano/cnode/logs/node0.json",
---
>       "scName": "/opt/cardano/cnode/logs/node-3.json",

Stop the old cardano-node service:

sudo systemctl stop cnode.socket
sudo systemctl stop cnode.service

Delete /etc/systemd/system/cnode.service and /etc/systemd/system/cnode.socket if they exist.

Create the following config and be sure to adjust the path for your system (for example jky should be your username):

$ cat /etc/systemd/system/cnode@.service
[Unit]
Description=Cardano Node %i
Requires=cnode@%i.socket
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
Restart=always
RestartSec=10000
User=jky
LimitNOFILE=1048576
WorkingDirectory=/opt/cardano/cnode/scripts
ExecStart=/home/jky/.local/bin/cardano-node run --topology /opt/cardano/cnode/files/topology.json --config "/opt/cardano/cnode/files/config-%i.json" --database-path "/opt/cardano/cnode/db-%i"
ExecStop=/bin/bash -l -c "exec kill -2 $(ps -ef |  grep "[c]ardano-node.*/opt/cardano/cnode/files/config-%i.json" | tr -s ' ' | cut -d ' ' -f2) &>/dev/null"
KillSignal=SIGINT
SuccessExitStatus=143
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=cnode
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target

$ cat /etc/systemd/system/cnode@.socket
[Unit]
Description=Socket for cardano node %i

[Socket]
ListenStream=/opt/cardano/cnode/sockets/node-%i.socket
Service = cnode@%i.service

[Socket]
ListenStream = 0.0.0.0:6000
ReusePort = yes
Service = cnode@%i.service
NoDelay = yes

[Socket]
ListenStream = 6000
ReusePort = yes
Service = cnode@%i.service
NoDelay = yes

[Install]
WantedBy = sockets.target

Start the new service:

for x in 1 2 3; do
  sudo systemctl start cnode@$x.socket
  sudo systemctl start cnode@$x.service
done

Wait for all the nodes to sync.

Once they are done, you can run cardano-ping -h localhost -p 6000 to connect to a random node.

To figure out which node it connected to, connect strace to each of the nodes in separate terminals like this:

sudo strace -p <pid> -f -e trace=accept4

Then run cardano-ping -h localhost -p 6000 multiple times to observe that connections are balanced randomly across all the nodes.

The following image shows example output as evidence this is working:

[newhoggy]

Troubleshooting

You can run journalctl -u cnode@n.service to check if your node failed to start where n is the number of the node you want to check.

The logs in /opt/cardano/cnode/logs/ may also be helpful.

[newhoggy]

Configuring

Currently cardano-node will want to three sockets: IPv4, IPv6 and Unix domain socket.

You will need to make sure that you have each one of these configured in systemd socket file. You can also have a mix of socketed activation and regular sockets, but you need to make sure for each and every kind of socket that cardano-node wants, you configure either socket activation socket or regular socket, but not neither and not both.

[johnalotoski]

This is being tested, thank you @newhoggy!

[johnalotoski]

Working great, thank you @newhoggy!