search

bitraf / infrastructure

Infrastructure stuff for Bitraf's sysadmins
1 stars 3 forks source link

bitnode - sometimes the disk drive fill up #134

Open tingox opened 5 years ago

tingox commented 5 years ago

Now and then (not very often) the hard disk drive on bitnode fills up. It looks like this:

tingo@bitnode:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   23G     0 100% /
devtmpfs        985M     0  985M   0% /dev
tmpfs           989M  4.0K  989M   1% /dev/shm
tmpfs           989M  105M  884M  11% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           989M     0  989M   0% /sys/fs/cgroup
tmpfs           198M     0  198M   0% /run/user/1009
tmpfs           198M     0  198M   0% /run/user/1004

this is unfortunate, services might stop because they run out of disk space when this happens. We should set up better monitoring, or find the root cause and fix it.

tingox commented 5 years ago

This time, trygvis fixed the problem by cleaning.

tingox commented 4 years ago

disk full on bitnode again

tingo@bitnode:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   23G     0 100% /
devtmpfs        985M     0  985M   0% /dev
tmpfs           989M  4.0K  989M   1% /dev/shm
tmpfs           989M  102M  888M  11% /run
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           989M     0  989M   0% /sys/fs/cgroup
tmpfs           198M     0  198M   0% /run/user/1004
tmpfs           198M     0  198M   0% /run/user/1009

I cleaned apt cache with sudo apt-get clean which helped a little.

tingox commented 4 years ago

I made a backup of the two stopped containers (mysql and wordpress) and then removed them via sudo lxc-destroy -n containername. Now disk space is a bit better

tingo@bitnode:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   20G  3.1G  87% /

Perhaps we should upgrade to more disk space?

tingox commented 2 years ago

Happened tonight again. The disk drive on bitnode was full. Fallout: postegresql stopped, and the wiki (mediawiki) stopped working because it had no databse. After I stopped and restarted the mediawiki container the disk drive is back to a bit over 50% full again?

tingo@bitnode:~$ date;df -h /
Tue 11 Oct 2022 09:20:54 PM CEST
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        24G   13G  9.9G  56% /

Hard to believe. Anyway, after starting postgresql service again, the wiki is up and running.

tingox commented 2 years ago

bitnode was resized yesterday. Now the disk drive is bigger

tingo@bitnode:~$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        79G   13G   63G  17% /

this helps a lot.