devnet: running a second rmb instance, breaks the first

[coesensbert]

https://github.com/threefoldtech/grid_deployment/tree/development/docker-compose/devnet

Upon running a second instance of the grid backend (complete stack, including rmb) I notice breaking relay.dev.grid.tf .

What happens

• The devnet backend hosted on *.dev.grid.tf is running fine, workload details load when selecting a node at https://dashboard.dev.grid.tf/explorer/nodes
• Start a second full backend at *.bknd1.dev.ava.tf
• Workload details load fine at https://dashboard.bknd1.dev.ava.tf/explorer/nodes
• Workload details stop loading at https://dashboard.dev.grid.tf/explorer/nodes -> https://gridproxy.dev.grid.tf/nodes/15/statistics times out
• Error at gridproxy.dev.grid.tf k8s pod:

  2023-11-24T10:19:31Z error failed to get get node statistics from relay: context deadline exceeded
  2023-11-24T10:19:31Z error failed to encode return object error="http: Handler timeout"

Then

Upon restarting the k8s gridproxy pod, https://dashboard.dev.grid.tf/explorer/nodes workload details load again. But workload details stop loading at https://dashboard.bknd1.dev.ava.tf/explorer/nodes (with same error in gridproxy, but the one serving ava.tf)

Reproducible every time

[muhamadazmy]

I did some debugging and find out the new relay at relay.bknd1.dev.grid.tf can't reach the internet completely (can't even ping google or 1.1.1.1)

This makes it impossible for that relay to do federation correctly and forward the message to the proper relay.

I will leave the issue open until resolved

[coesensbert]

ok so recently removed the static networking and started working with hostnames, this made docker deploy into the 192.168.0.0/16 range, which was not masqueraded by nftables, now it is:

  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 172.16.0.0/12 masquerade
    ip saddr 192.168.0.0/16 masquerade
    ip6 saddr 2001:db8:1::/64 masquerade
  }

[coesensbert]

both instances ran the same mnemonic, resolved