search

enram / data-repository

Data quality assessment
https://enram.github.io/data-repository/
MIT License
3 stars 1 forks source link

File transfer: make sure the EC2 machine doesn't get out of disk space #58

Closed niconoe closed 4 years ago

niconoe commented 4 years ago

... that happened on June 20th, 2020, blocking further transfers. Fortunately we got informed thanks to a mail alert.

It appears a local copy of files happen on this machine, investigate if this is really necessary?

niconoe commented 4 years ago

I'd like to self-assign this issue, but can't now. @peterdesmet : can you give me some rights on this repo? Thanks!

niconoe commented 4 years ago

Update: disk appears full not because of size, but because the number of inodes gets exceeded. A reboot temporary solved it (tmp files deleted?):

$ df -i
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
udev           124436    357 124079    1% /dev
tmpfs          126758    446 126312    1% /run
/dev/xvda1     786432 786401     31  100% /
tmpfs          126758      1 126757    1% /dev/shm
tmpfs          126758      3 126755    1% /run/lock
tmpfs          126758     16 126742    1% /sys/fs/cgroup
tmpfs          126758      4 126754    1% /run/user/1000
niconoe commented 4 years ago

Update: I was able to remove a tons of old kernel sources trees in /usr/src following this method, there's now a tons of free inodes: https://ubuntugenius.wordpress.com/2011/01/08/ubuntu-cleanup-how-to-remove-all-unused-linux-kernel-headers-images-and-modules/

Next step: reboot the instance to make sure it still has the needed kernel (but all looks right: menu.lst, vmlinuzimage, ...) => DONE

peterdesmet commented 4 years ago

Thanks for investigating Nico, you now have maintainer rights on this repo

adokter commented 4 years ago

I've run into inodes issues too recently on my local machine, because of the large number of small size when storing vp files for each radar over a long time series. I had to reformat the disk and assign more inodes to fix it:

mkfs.ext4 -T news /dev/device

formatting with this statement gives you one inode for every 4096 bytes, instead of default 16384 bytes.

niconoe commented 4 years ago

Thanks @adokter, in that case it was just old linux (kernel) sources in /usr/src that were taking 90%+ of inodes, but that's good to know for the future!

niconoe commented 4 years ago

TODO:

Use AWS features to make sure I am informed by email next time the disk space/number of free inodes get low.

niconoe commented 4 years ago

Server back and running, and the appropriate CloudWatch have been set so I'll be informed the next time we get low on disk space or free inodes.