keroscarel / s3backer

Automatically exported from code.google.com/p/s3backer
GNU General Public License v2.0
0 stars 0 forks source link

s3backer never releases memory and reach to peak after 10 -15 days. #47

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago
What steps will reproduce the problem?
1.Remounting s3backer 
2.
3.

What is the expected output? What do you see instead?
ps -uH `pid of s3backer`

O/P->

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:00 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:00 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:00 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer
root      4485  0.0  2.3 878512 88808 ?        Ssl  01:42   0:01 
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http quarem3-production-backup /s3backer

--------------------------------------------------------------------------

lsof -p `pidof s3backer`
o/p->

COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
s3backer 4485 root  cwd    DIR             202,65     4096        2 /
s3backer 4485 root  rtd    DIR             202,65     4096        2 /
s3backer 4485 root  txt    REG             202,65   347828   294400 
/usr/local/s3backer/bin/s3backer
s3backer 4485 root  mem    REG             202,65    91096   294091 
/lib64/libz.so.1.2.3
s3backer 4485 root  mem    REG             202,65   224328   267630 
/usr/lib64/libssl3.so
s3backer 4485 root  mem    REG             202,65   317168   294115 
/lib64/libldap-2.4.so.2.5.6
s3backer 4485 root  mem    REG             202,65    63304   285445 
/lib64/liblber-2.4.so.2.5.6
s3backer 4485 root  mem    REG             202,65   386040   280450 
/lib64/libfreebl3.so
s3backer 4485 root  mem    REG             202,65    43392   287443 
/lib64/libcrypt-2.12.so
s3backer 4485 root  mem    REG             202,65  1286744   265631 
/usr/lib64/libnss3.so
s3backer 4485 root  mem    REG             202,65   243096   287440 
/lib64/libnspr4.so
s3backer 4485 root  mem    REG             202,65    17096   285261 
/lib64/libplds4.so
s3backer 4485 root  mem    REG             202,65   177952   285270 
/usr/lib64/libnssutil3.so
s3backer 4485 root  mem    REG             202,65    21256   283747 
/lib64/libplc4.so
s3backer 4485 root  mem    REG             202,65   183896   285435 
/usr/lib64/libsmime3.so
s3backer 4485 root  mem    REG             202,65   108728   285457 
/usr/lib64/libsasl2.so.2.0.23
s3backer 4485 root  mem    REG             202,65   124624   285408 
/lib64/libselinux.so.1
s3backer 4485 root  mem    REG             202,65    17256   285418 
/lib64/libcom_err.so.2.1
s3backer 4485 root  mem    REG             202,65   915736   285419 
/lib64/libkrb5.so.3.3
s3backer 4485 root  mem    REG             202,65    12592   280447 
/lib64/libkeyutils.so.1.3
s3backer 4485 root  mem    REG             202,65   181632   285417 
/lib64/libk5crypto.so.3.1
s3backer 4485 root  mem    REG             202,65    46336   285416 
/lib64/libkrb5support.so.0.1
s3backer 4485 root  mem    REG             202,65   272360   285420 
/lib64/libgssapi_krb5.so.2.2
s3backer 4485 root  mem    REG             202,65   167648   267620 
/lib64/libexpat.so.1.5.2
s3backer 4485 root  mem    REG             202,65  1953536   294417 
/usr/lib64/libcrypto.so.1.0.1e
s3backer 4485 root  mem    REG             202,65   444040   294443 
/usr/lib64/libssl.so.1.0.1e
s3backer 4485 root  mem    REG             202,65   164024   294470 
/usr/lib64/libssh2.so.1.0.1
s3backer 4485 root  mem    REG             202,65   343544   294489 
/usr/lib64/libcurl.so.4.1.1
s3backer 4485 root  mem    REG             202,65   156872   282897 
/lib64/ld-2.12.so
s3backer 4485 root  mem    REG             202,65  1908792   282898 
/lib64/libc-2.12.so
s3backer 4485 root  mem    REG             202,65    22536   282900 
/lib64/libdl-2.12.so
s3backer 4485 root  mem    REG             202,65   141576   265675 
/lib64/libpthread-2.12.so
s3backer 4485 root  mem    REG             202,65   586000   264533 
/usr/lib64/libsqlite3.so.0.8.6
s3backer 4485 root  mem    REG             202,65    47064   282907 
/lib64/librt-2.12.so
s3backer 4485 root  mem    REG             202,65   113952   265677 
/lib64/libresolv-2.12.so
s3backer 4485 root  mem    REG             202,65   209120   268225 
/lib64/libidn.so.11.6.1
s3backer 4485 root  mem    REG             202,65   150712   268884 
/usr/lib64/libnsspem.so
s3backer 4485 root  mem    REG             202,65    10352   268891 
/usr/lib64/libnsssysinit.so
s3backer 4485 root  mem    REG             202,65   254008   267587 
/usr/lib64/libsoftokn3.so
s3backer 4485 root  mem    REG             202,65    27424   265111 
/lib64/libnss_dns-2.12.so
s3backer 4485 root  mem    REG             202,65    65928   265113 
/lib64/libnss_files-2.12.so
s3backer 4485 root  mem    REG             202,65   221728   283744 
/lib64/libfuse.so.2.8.3
s3backer 4485 root    0u   CHR                1,3      0t0     3593 /dev/null
s3backer 4485 root    1u   CHR                1,3      0t0     3593 /dev/null
s3backer 4485 root    2u   CHR                1,3      0t0     3593 /dev/null
s3backer 4485 root    3u  unix 0xffff8800aaa7d3c0      0t0 10468420 socket
s3backer 4485 root    4u   CHR             10,229      0t0     6315 /dev/fuse
s3backer 4485 root    5u  IPv4           10523049      0t0      TCP 
ip-10-84-201-9.ec2.internal:38000->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root    6u   REG             202,65     9216   285288 
/etc/pki/nssdb/cert9.db
s3backer 4485 root    7u   REG             202,65    11264   285348 
/etc/pki/nssdb/key4.db
s3backer 4485 root    8u  IPv4           10523311      0t0      TCP 
ip-X.X.X.X.ec2.internal:39297->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root    9u   REG             202,65      512    12333 
/var/tmp/etilqs_PCnEh0NTRnLafgM (deleted)
s3backer 4485 root   10u   REG             202,65        0    12402 
/var/tmp/etilqs_kwaINYMcSch8taP (deleted)
s3backer 4485 root   11u   REG             202,65     2048    17431 
/var/tmp/etilqs_14hb2M6Espx6yd3 (deleted)
s3backer 4485 root   12u  IPv4           10522912      0t0      TCP 
ip-X-X-X-XX.ec2.internal:39213->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root   13u   REG             202,65      512    17625 
/var/tmp/etilqs_dmz8dBtg0Wip1X2 (deleted)
s3backer 4485 root   14u   REG             202,65        0    17626 
/var/tmp/etilqs_6h9jnSb83YWnVTZ (deleted)
s3backer 4485 root   15u   REG             202,65     2048    17737 
/var/tmp/etilqs_OVTKh6mwgpGsooP (deleted)
s3backer 4485 root   16u  sock                0,6      0t0 10523297 can't 
identify protocol
s3backer 4485 root   17u  IPv4           10523326      0t0      TCP 
ip-X-X-X-X.ec2.internal:39301->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root   18u  IPv4           10523868      0t0      TCP 
ip-X-X-X-X.ec2.internal:60583->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root   19u  IPv4           10523314      0t0      TCP 
ip-X-X-X-X.ec2.internal:39299->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)
s3backer 4485 root   20u  IPv4           10523050      0t0      TCP 
ip-X-X-X-X.ec2.internal:39249->XXXXXXXX.amazonaws.com:https (CLOSE_WAIT)

What version of the product are you using? On what operating system?

s3backer version 1.3.7 (r496)

Please provide any additional information below.

*M/C:- Amazon Medium instance with 4GB RAM
*OS:-CentOS Linux release 6.0 X86_64 
*Configuration:-
/usr/local/s3backer/bin/s3backer --blockSize=64k --size=50g --ssl --vhost 
--listBlocks --debug-http XXXXXX-XXXXX-backup /s3backer

Original issue reported on code.google.com by rakesh.p...@gmail.com on 5 May 2014 at 1:16

GoogleCodeExporter commented 8 years ago
It looks like s3backer is using 2.3% of memory. So what exactly is the problem?

Original comment by archie.c...@gmail.com on 5 May 2014 at 1:47

GoogleCodeExporter commented 8 years ago
We have a daily backup running and copies file to bucket everyday thru rysc 
,Now it is using 2.3% but it will grow to peak after some days(2 GB and above) 
without coming back to normal state.
Finally s3backer process will go to 'D' state ,and there is no other option 
except rebooting the server.

Original comment by rakesh.p...@gmail.com on 5 May 2014 at 1:54

GoogleCodeExporter commented 8 years ago
OK, sounds like there is a memory leak, at least in your particular setup.

If you have a C developer handy you might try running s3backer under gdb and 
once it has filled up memory try to see if you can determine what is being 
leaked.

Original comment by archie.c...@gmail.com on 5 May 2014 at 2:03

GoogleCodeExporter commented 8 years ago
I don't have C developer nor me :( , Do you find any wrong values in my 
configuration? , How others are using in there setup.
Does s3backer really has memory leak issue with centos 6 setup?(Pls provide any 
related links).. If I update my OS version will it resolves the problem?

Original comment by rakesh.p...@gmail.com on 5 May 2014 at 2:18

GoogleCodeExporter commented 8 years ago
If you are only using the command-line flags that are shown then your 
configuration is fine. If you were using a very large, in-memory block cache 
then that could cause those symptoms, but it appears you are using the default 
block cache size of 1000, so that would not be it (64KB x 1000 = 64MB max).

Updating the O/S could possibly fix the problem if the problem is within one of 
the libraries that s3backer links to (e.g., libcurl, libopenssl, etc.). That's 
just speculation however. Worth a try if you can do it.

Original comment by archie.c...@gmail.com on 5 May 2014 at 2:24

GoogleCodeExporter commented 8 years ago
Yes I am using the same command line configuration, Let you know the status 
after OS up gradation.

Original comment by rakesh.p...@gmail.com on 5 May 2014 at 7:19

GoogleCodeExporter commented 8 years ago
Hey Archie,
I have a doubt, Now s3backer and s3bucket is already mounted in production AWS, 
can I install and configure same s3backer and s3bucket into my local machine?
I think it is like NFS where we can mount multiple times in different m/c's 
,Please let  me know if this works and doesn't harm production server.

Original comment by rakesh.p...@gmail.com on 6 May 2014 at 9:31

GoogleCodeExporter commented 8 years ago
I'll assume you are using a normal "upper" filesystem like ext4. If so then you 
must NOT mount s3backer from two different systems at the same time. Only a few 
very specialized "upper" filesystems support doing that.

s3backer is just like a hard disk.. you wouldn't connect the same hard disk to 
two different machines at the same time either, unless those machines had some 
special software.

Original comment by archie.c...@gmail.com on 6 May 2014 at 2:12