mainnet: change layout

[yakimant]

Change layout:

1. stop BNs and ELs
2. run asnible playbook
3. (geth) stop geth, replace geth db, start
4. start a stable BN (check ports), start new BNs and sync.sh them

Cleanup checklist:

1. sudo rm -rvf /etc/consul/*_0* /etc/consul/service_geth_mainnet_node.json /etc/consul/service_geth_mainnet_exporter.json
2. sudo systemctl reload consul
3. sudo systemctl stop 'beacon.*\-0'
4. sudo systemctl disable update-beacon-node-mainnet-unstable-02.timer update-beacon-node-mainnet-unstable-01.timer update-beacon-node-mainnet-testing-01.timer update-beacon-node-mainnet-stable-01.timer resync-beacon-node-mainnet-unstable-02.timer resync-beacon-node-mainnet-testing-01.timer beacon-node-mainnet-stable-01.service beacon-node-mainnet-testing-01.service beacon-node-mainnet-unstable-01.service beacon-node-mainnet-unstable-02.service monitor-beacon-node-mainnet-stable-01.service monitor-beacon-node-mainnet-testing-01.service monitor-beacon-node-mainnet-unstable-01.service monitor-beacon-node-mainnet-unstable-02.service
5. sudo rm -rvf /etc/systemd/system/*beacon*-0*
6. sudo systemctl daemon-reload
7. docker stop geth-mainnet geth-mainnet-exporter
8. docker system prune -a --volumes
9. sudo rm -rvf /var/log/service/beacon-node-mainnet-*0* /var/log/docker/geth-mainnet-exporter /var/log/docker/geth-mainnet-node
10. sudo rm -rvf /docker/geth-mainnet/ /data/beacon-node-mainnet-*0*
11. sudo rm -rvf /etc/sudoers.d/*beacon*-0*

[yakimant]

Issues:

• [x] geth-02: testing BN consul check missing - it's Ok, public_api
• [x] geth-02: stable graphs missing
• [x] geth-02: geth metrics exporter missing

[yakimant]

• [ ] erigon nodes are not syncing:

  
  ❯ ./rpc.sh eth_blockNumber
  bash: warning: setlocale: LC_ALL: cannot change locale (en_GB.UTF-8)
  {
  "jsonrpc": "2.0",
  "id": 1,
  "result": "0x0"
  }

❯ sudo du -sh erigon-mainnet-* 14M erigon-mainnet-stable 12M erigon-mainnet-unstable

https://github.com/status-im/infra-nimbus/issues/211

[jakubgs]

I think we can merge the layout changes even if some nodes are not fully synced, and follow that up int he issue(which should be linked in this PR and commit).