gravitl / netmaker

Netmaker makes networks with WireGuard. Netmaker automates fast, secure, and distributed virtual networks.
https://netmaker.io
Other
9.5k stars 552 forks source link

Memory leak #533

Closed ethanfowler closed 2 years ago

ethanfowler commented 2 years ago

Hi,

I have the standard install running on an Amazon EC2 t2.micro. Every couple of days it goes down, stops responding to anything, and I have to restart. I added a Cloudwatch agent in order to monitor memory utilisation; looks like there's a leak that's bringing the instance to its knees.

image

ethanfowler commented 2 years ago

There is definitely a memory leak making hosting on small (t2.micro) servers impossible without hacks. My current hack is a cron job that checks memory availability and restarts the docker containers if it is low.

The below assumes that your host is Ubuntu 20.04 and docker-compose.yml is in /root/.; adapt as necessary.

sudo crontab -e and add to the end of the file:

*/5 * * * * /root/restart_if_memory_leak.sh

Create /root/restart_if_memory_leak.sh and populate it with:

#!/bin/bash

available_memory=$(free -m | awk '/Mem/ {print $7}')
echo "Free memory: ${available_memory}MB"
if [[ $available_memory -lt 200 ]]; then
    echo "Restarting netmaker..."
    pushd /root
    docker-compose restart
    popd
    echo "Done"
fi

I added this cron job to my instance after another outage on 12/08, and it seems to be keeping things alive: image

afeiszli commented 2 years ago

fixed in 0.9.2