search

CuiMingFu / zumastor

Automatically exported from code.google.com/p/zumastor
0 stars 1 forks source link

Zumastor might consume all the free space in /var/run #148

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
What steps will reproduce the problem?
1. Install Zumastor and configure replication
2. Replication starts

What is the expected output? What do you see instead?

All the free space in /var/run is used, therefore other daemons cannot start. 

This is on Ubuntu Hardy i386, 4GB RAM, 2 x 1TB volumes, one of them empty,
the other one 100GB of data (and only about 20GB being replicated). Data is
being sent over the LAN with no compression. Zumastor 0.8.0-r1659.

I had never seen this error with Ubuntu Gutsy. Also, I feel like Zumastor
was faster on Gutsy. I've not used Hardy long enough to tell if this
happens often or is a specific issue.

Original issue reported on code.google.com by pgqui...@gmail.com on 26 May 2008 at 2:56

GoogleCodeExporter commented 9 years ago 
The problem is not inode-exhaustion, by the way:

root@maquina1:/var/run/zumastor# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/cciss/c0d0p1    4685824   46114 4639710    1% /
varrun                219929    6849  213080    4% /var/run
varlock               219929       5  219924    1% /var/lock
udev                  219929    3092  216837    2% /dev
devshm                219929       1  219928    1% /dev/shm
lrm                   219929      18  219911    1%
/lib/modules/2.6.24-16-zumastor/volatile
/dev/cciss/c0d1p1    61038592  522744 60515848    1% /media/justreplicated
/dev/mapper/zumavm   30523392      12 30523380    1% 
/var/run/zumastor/mount/zumavm
/dev/mapper/zumaf    30539776  529310 30010466    2% 
/var/run/zumastor/mount/zumaf

root@maquina1:/var/run/zumastor# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p1      71G  1.5G   66G   3% /
varrun                1.7G  1.7G     0 100% /var/run
varlock               1.7G     0  1.7G   0% /var/lock
udev                  1.7G  108K  1.7G   1% /dev
devshm                1.7G     0  1.7G   0% /dev/shm
lrm                   1.7G   38M  1.7G   3% 
/lib/modules/2.6.24-16-zumastor/volatile
/dev/cciss/c0d1p1     924G  265G  660G  29% /media/justreplicated
/dev/mapper/zumavm    462G  199M  462G   1% /var/run/zumastor/mount/zumavm
/dev/mapper/zumaf     463G   99G  364G  22% /var/run/zumastor/mount/zumaf

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:01

GoogleCodeExporter commented 9 years ago 
RAM is not the problem, either:

root@maquina1:/var/run/zumastor# free
             total       used       free     shared    buffers     cached
Mem:       3499524    2569672     929852          0     627940    1350008
-/+ buffers/cache:     591724    2907800
Swap:      3220992    1779224    1441768

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:02

GoogleCodeExporter commented 9 years ago 
Here is a snippet of /var/log/samba/log.smbd showing what happens due to 
/var/run
being full:

[2008/05/26 04:47:51, 0]
/build/buildd/samba-3.0.28a/source/lib/pidfile.c:pidfile_create(133)
  ERROR: can't write to file /var/run/samba/smbd.pid: No space left on device

As smbd does not start, Samba does not work properly and shared folders are not
accessible in this machine.

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:06

GoogleCodeExporter commented 9 years ago 
Can you identify any large files?  e.g.
  find /var/run -size +1M -ls

Original comment by daniel.r...@gmail.com on 26 May 2008 at 3:13

GoogleCodeExporter commented 9 years ago 
Thinking about it, these are the exact steps which led me to this failure:

1. Samba is stopped (i. e. there are no pidfiles in /var/run)
2. Install Zumastor and configure replication
3. Dump a lot of data to a Zumastor volume, replication starts
4. Start Samba
5. Profit! errr Fail! :-)

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:17

GoogleCodeExporter commented 9 years ago 
Can you perform `du -xsh /var/run/zumastor/*`
Also, I am guessing you are using the 2.6.24 kernel?

Original comment by compbr...@gmail.com on 26 May 2008 at 3:35

GoogleCodeExporter commented 9 years ago 
root@maquina1:~# du -xsh /var/run/zumastor/*
0       /var/run/zumastor/agents
0       /var/run/zumastor/cron
16K     /var/run/zumastor/mount
0       /var/run/zumastor/running
0       /var/run/zumastor/servers
0       /var/run/zumastor/snapshot
4.0K    /var/run/zumastor/zumaf-server.pid
4.0K    /var/run/zumastor/zumavm-server.pid

Yes, I'm running 2.6.24:

root@maquina1:~# uname -a
Linux maquina1 2.6.24-16-zumastor #1 SMP Sat Apr 26 05:48:44 UTC 2008 i686 
GNU/Linux

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:45

GoogleCodeExporter commented 9 years ago 
Dan, there are many many large files (the data I dumped). I'm waiting for the
replication to finish to see if it "magically" frees space.

Original comment by pgqui...@gmail.com on 26 May 2008 at 3:47

GoogleCodeExporter commented 9 years ago 
Hang on, try the commandline
  find /var/run -xdev -size +1M -ls
That will find large files in /var/run itself and not in filesystems
mounted in /var/run.
Maybe a mount failed, and the data went into /var/run and not to
a mounted volume?

Original comment by daniel.r...@gmail.com on 26 May 2008 at 4:42

GoogleCodeExporter commented 9 years ago 
I have been unable to reproduce this behavior. I suggest this issue be closed 
and
marked as invalid.

Original comment by pgqui...@gmail.com on 2 Jun 2008 at 3:41

GoogleCodeExporter commented 9 years ago 
ok

Original comment by daniel.r...@gmail.com on 2 Jun 2008 at 3:48