search

TritonDataCenter / smartos-live

For more information, please see http://smartos.org/ For any questions that aren't answered there, please join the SmartOS discussion list: https://smartos.topicbox.com/groups/smartos-discuss
1.59k stars 249 forks source link

Excessive Logging in /var/adm/messages caused full /var and server to hang #294

Open kfr- opened 10 years ago

kfr- commented 10 years ago

/var/adm/messages fills up with 50+ GB of recurring entries.

This has happened to me twice already on different servers.

[root@bvc-srv1 ~]# uname -a SunOS bvc-srv1 5.11 joyent_20131102T215831Z i86pc i386 i86pc

head /var/adm/messages 2014-01-16T20:59:18.472054+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.472100+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.472981+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x130076 2014-01-16T20:59:18.473029+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.473118+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.473513+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.474012+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x130076 2014-01-16T20:59:18.474056+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076 2014-01-16T20:59:18.493642+00:00 changeme kvm: [ID 987709 kern.info] unimplemented perfctr wrmsr: 0xc0010000 data 0x530076

-rw-r--r-- 1 root root 18G Dec 20 20:45 messages.3 -rw-r--r-- 1 root root 254M Dec 27 20:45 messages.2 -rw-r--r-- 1 root root 1.3G Jan 3 20:47 messages.1 -rw-r--r-- 1 root root 31K Jan 10 20:47 messages.0 -rw-r--r-- 1 root root 30G Jan 17 18:14 messages

kfr- commented 10 years ago

I think i found a work around. Well, kind of

cp /etc/logadm.conf /opt/custom/etc/

changed /var/adm/messages -C 4 -a 'kill -HUP cat /var/run/*syslog*.pid' to /var/adm/messages -C 4 -s 2g -a 'kill -HUP cat /var/run/*syslog*.pid'

mount -F lofs /opt/custom/etc/logadm.conf /etc/logadm.conf