gravitl / netmaker

Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
https://netmaker.io
Other
9.5k stars 552 forks source link

[Bug]: High CPU load crashing server with internet gateway on AWS #2602

Open bpmarine opened 1 year ago

bpmarine commented 1 year ago

Contact Details

bpmarine@gmail.com

What happened?

Testing two different instances of AWS Lightsail servers, setting up as default CE installs with custom domains using the latest v 0.21 quick-install script.

Single host installed by default during the installation. Added egress gateway to this host with internet gateway 0.0.0.0 for exit and no other networks. Single external client config added.

Connecting to the server from a Pixel 6 and the wireguard app. Observed no DNS resolution so I set default DNS as 9.9.9.9 to test. Resolved properly with this set.

While the external client is connected I'm seeing 100% CPU utilisation on the server. Available burst time drops to 0%. After several hours the server stops responding until rebooted from the Lightsail console.

Tested two different Lightsail instances, same results. No other configuration tested.

Version

v0.21.0

What OS are you using?

Linux

Relevant log output

No response

Contributing guidelines

bpmarine commented 1 year ago

Actually, I now see that the CPU usage while I'm connected is within 0-20%, very low. Spikes are occurring at other times and crashing the server

harry-hart commented 1 year ago

I just experienced the same thing on a Lightsail server running the netmaker server. Huge spike in cpu load, stopped responding to the network. I had to force stop it and then restart the server. It is a bare Ubuntu server with just Netmaker installed so there is unlikely any other causes.

abhishek9686 commented 1 year ago

@bpmarine @harry-hart what's the memory and cpu configuration of your server?

harry-hart commented 1 year ago

1 GB RAM, 2 vCPUs on a lightsail server. In normal usage it hovers around 1% usage, but was sustained bursting to 80% when the "bug" occured.

Server was version: v0.20.6

harry-hart commented 1 year ago

image

This is the graph of cpu usage. The drop was when I force shutdown.