search

gravitl / netmaker

Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
https://netmaker.io
Other
9.4k stars 547 forks source link

[Bug]: Empty ip address in external client #2194

Closed uetaaam closed 1 year ago

uetaaam commented 1 year ago

Contact Details

No response

What happened?

, I've been using netmaker 0.17 on my other server without any problems. Now I have installed the latest version using the install script on small OVH vps. My default node already has ingress enabled after installation and my other nodes can ping each other. I have problem with external clients - when i try to add new config for external client i always get blank ip address field. And when I download this config it always contains this one ip address [Interface] Address = 10.101.255.254/32 And I cannot ping any of the nodes. What am I doing wrong?

Version

v0.18.5

What OS are you using?

No response

Relevant log output

No response

Contributing guidelines

mattkasun commented 1 year ago

what is the cidr of your network. If it is 10.101.0.0/16 the ipaddress for you extclient is correct.

uetaaam commented 1 year ago

what is the cidr of your network. If it is 10.101.0.0/16 the ipaddress for you extclient is correct.

Yes, now I see that it is correct address but I still cant get proper connection from ext clients to any of my nodes despite the fact that nodes can ping each other

mattkasun commented 1 year ago

what os is the ingress node running? Is IP forwarding enabled on the ingress node?

uetaaam commented 1 year ago

what os is the ingress node running? Is IP forwarding enabled on the ingress node?

I am using ubuntu 22.04. I've updated compose file to gravitl/netmaker:develop and image: gravitl/netmaker-ui:develop and now everything is working fine

uetaaam commented 1 year ago

what os is the ingress node running? Is IP forwarding enabled on the ingress node?

I am using ubuntu 22.04. I've updated compose file to gravitl/netmaker:develop and image: gravitl/netmaker-ui:develop and now everything is working fine

@mattkasun Actually not completly fine, I can only connect to some nodes from my external clients. This seems to be random as the servers I can connect to from external clients change over time. In the screenshot below you can see this behaviour. I have a few dozen nodes and a few external clients, on all of them the list of nodes with proper connectivity is the same.

image

alcroito commented 1 year ago

I'm not sure if you have the same problem that i had, but i had some similar symptoms. My ingress node was running netclient on an ubuntu 22.04 host (not in a docker container). The iptables forwarding rules created by netmaker were not sufficient for forwarding all packets, if the default policy of the FORWARD chain in the filter table was set to DROP. This caused packets to be routed only half way, so the pings were sent, but the replies were not forwarded back. The policy was being set to DROP by the docker service (it's documented as such on their website). Changing the policy to ACCEPT helped. That's not the best solution, but it was fine for me.

mattlathrop commented 1 year ago

I believe this is happening to me as well. @alcroito I tried running sudo iptables -P FORWARD ACCEPT but I've got the same issue. Is that what you did to solve it?

alcroito commented 1 year ago

Sorry, it was late when I wrote the last reply, my issue was with an egress node with packets not being returned from an egress network. Essentially i ran that command, yes, i just disabled the docker systemd service on the egress node, so it doesn't modify the forward policy to drop.

Regarding your problem, i'd first check if the ingress node can ping the nodes you're trying to ping from an external client. I saw cases where the wireguard handshake between nodes (as shown by sudo wg show) was not updated for a long time, which means no connection was possible. I never figured out why that happened, even with wireguard kernel logs enabled and wireshark logs, the handshake packets were sent from both sides, but never received by the recipients.

Also if possible, I'd also try to completely clear all iptable rules on the ingress / target nodes for the nat and filter tables, and restart then netclient on both nodes, so it re-adds the forwarding rules, and then try pinging again.

Running wireshark was helpful to see at which point did packets got lost sudo tshark -i any "icmp".

And iptables has a packet counter next to each rule or chain, which gets incremented every time a packet is received. Watching those numbers after calling ping can help detect if a packet got processed by the wrong rule, or dropped.

watch -n 2 sudo iptables -L -v -n -t filter --line-number. Look for increases in the netmaker and FORWARD chains.

watch -n 2 sudo iptables -L -v -n -t nat --line-number. Look for increases in the netmaker MASQUERADE rule.

mattkasun commented 1 year ago

fixed in v0.18.7