search

cloyne / network

5 stars 5 forks source link

address server1 disk usage #106

Closed ck2qsuZT closed 7 years ago

ck2qsuZT commented 7 years ago

There has been an email from server 1s chron deamon every hour for the past two days now. the initial message is 92% disk usage on the boot partition, not sure what's going on with that but the drive has plenty of free space in case the boot partition needs to be expanded for some reason.

mitar commented 7 years ago

Have you even looked what is using the space? Ccc.

ck2qsuZT commented 7 years ago

looking at it now

ck2qsuZT commented 7 years ago

why are there so many kernel versions...

mitar commented 7 years ago

Autoupgrade.

Just remove old ones. :-)

BTW, the same issue is on server2 as well.

ck2qsuZT commented 7 years ago

I was also trying something with this git issue by outlining my thought process, I wasn't really asking for help but I appreciate it =)

ck2qsuZT commented 7 years ago

just ran rm *xx-generic a bunch of times

where xx is the version number. made a mistake because initrd.img's last version was one less than the last version of everything else and accidentally deleted all server1's initrd images but I just coppied over the image from server2.

There's probably a better way to do this other than manually

Leaving open for a bit until I see if there's a better way to do it other than manually.

mitar commented 7 years ago

I was also trying something with this git issue by outlining my thought process, I wasn't really asking for help but I appreciate it =)

That is great. I jumped on only because you jumped to an invalid conclusion that it is a log rotate issue. If you would just write "disk is getting full" and then started investigating, I would not interfere. :-)

just ran rm *xx-generic a bunch of times

NOOOO. You should never ever remove system files on Debian manually. Use dpkg -S to find a package which installed them and then remove a package.

mitar commented 7 years ago

but I just coppied over the image from server2.

The same as above. Find a package of the file you removed by accident, and then do apt-get install --reinstall to get it back. Copying from other servers is dangerous.

ck2qsuZT commented 7 years ago

I know you shouldn't do it, but it's a fun test ground since server1 isn't really used. I like taking things apart =)

I know copying the initrd won't work, but I was curious what exactly it would do before properly fixing it =p It should not be done because server2 uses software RAID thus copying it over will unnecessarily load the mdm modual in init. besides that and more importantly, the initrd of server2 has the root partition on /dev/mapper/server2-vg-root instead of server1

mitar commented 7 years ago

I think VirtualBox is a great place to play with such things. :-)

ck2qsuZT commented 7 years ago

But where's the fire?

Yes, VirtualBox is a much better test ground, it allows fancy snapshots that undo any damage you may accidentally do. but sometimes its fun to climb without a safety harness =)

ck2qsuZT commented 7 years ago

Also, this stack exchange has more detailed answers on how to properly clean a debian boot partition

http://askubuntu.com/questions/89710/how-do-i-free-up-more-space-in-boot

The reason to not manually delete old files is because they are sometimes still referenced by other things. This thread is mostly things I have done in the past, my thought process during those times, what things worked/didn't because I can forsee other network managers doing the same thing as I did in my earlier years, and then crying horrid tears of frustration as everything stops working

ck2qsuZT commented 7 years ago

Now server1 is officially broken since the kernel version in use was actually older than the latest kernel image since the server was awaiting restart and I deleted the in use kernel manually. Now I'll document how to fix it =)

ck2qsuZT commented 7 years ago

If you've already rebooted and are sad because it doesn't boot anymore: http://askubuntu.com/questions/28099/how-to-restore-a-system-after-accidentally-removing-all-kernels

If you realized your mistake before rebooting and want to save yourself some pain there is a very relevant link: http://askubuntu.com/questions/561251/i-think-i-accidentally-removed-the-current-linux-kernel-when-trying-to-free-up-s

ck2qsuZT commented 7 years ago

This thread is making me sound insane...

ck2qsuZT commented 7 years ago

Also, accidentally ran the rm commands on both server1 and server2.... sometimes you play with fire and get burned... learning from mistakes is fun. Not a problem since the second link is helpful =p

mitar commented 7 years ago

BTW, you can always do mv instead of rm, test things, and then remove for real once you figure out that it works.