search

dashpay / dash

Dash - Reinventing Cryptocurrency
https://www.dash.org
MIT License
1.49k stars 1.2k forks source link

v0.12.0.47 - potential memory leak #568

Closed schinzelh closed 9 years ago

schinzelh commented 9 years ago

Hi,

i have noticed that memory consumption of dashd steadily grows from ~250M after t_startup + 5min

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
27397 dashd     30  10 1534.2m 255.8m 100.8m S   2.0  3.2   2:17.24 dashd

to 620M+ after t_startup + 4d

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
27397 dashd     30  10 1992.6m 624.6m  87.1m S   2.0  8.0 101:46.39 dashd

whereas v11 had a stable memory footprint.

Anything i can do to debug this?

schinzelh commented 9 years ago

Similar report at https://dashtalk.org/threads/v12-release.5888/page-34#post-64719

Anyone else seeing potential memory issues on your nodes? i had to stop and start a bunch yesterday as memory usage by dashd had been steadily climbing until i finally stopped it at around 480M. I restarted dash and they were all back at ~190MB. Now about 24hrs later I checked and they are all between 300 and 370MB currently. See the attached chart for one server with 512MB RAM and 512MB swap. I'll keep watching it but it seems to be creeping up again. The each have about 20-30 connections which is on the low side historically, so I don't think that's the cause. All are on v47.

image

minersday commented 9 years ago

Here's mem usage from couple of masternodes (weekly graph) 1: node1-memusage 2: node2-memusage 3: node3-memusage

minersday commented 9 years ago 
sudo pmap -x PROCESS
PROCESS:   dashd
Address           Kbytes     RSS   Dirty Mode  Mapping
00007f6990000000   56312   53656   53656 rwx--   [ anon ]
00007f69936fe000    9224       0       0 -----   [ anon ]
00007f6998000000   65536   65496   65496 rwx--   [ anon ]
00007f69a0000000   65536   65384   65384 rwx--   [ anon ]
00007f69a8000000   65532   65504   65504 rwx--   [ anon ]
00007f69abfff000       4       0       0 -----   [ anon ]
00007f69b0000000   65520   65504   65504 rwx--   [ anon ]
00007f69b3ffc000      16       0       0 -----   [ anon ]
00007f69b8000000   65536   65456   65456 rwx--   [ anon ]
00007f69be71d000    2124     680       0 r-xs- 000716.ldb
00007f69beb43000    2124     636       0 r-xs- 000714.ldb
00007f69bed56000    2124     748       0 r-xs- 000713.ldb
00007f69bef69000    2124     664       0 r-xs- 000712.ldb
00007f69bf38f000    2120     592       0 r-xs- 000709.ldb
00007f69bf5aa000    2128     456       0 r-xs- 000741.ldb
00007f69bf7be000    2084    1832       0 r-xs- 000427.ldb
00007f69bfbda000    2124     640       0 r-xs- 000711.ldb
00007f69c0000000     132      64      64 rwx--   [ anon ]
00007f69c0021000   65404       0       0 -----   [ anon ]
00007f69c4000000     292     252     252 rwx--   [ anon ]
00007f69c4049000   65244       0       0 -----   [ anon ]
00007f69c8000000    3932    1948    1948 rwx--   [ anon ]
00007f69c83d7000   61604       0       0 -----   [ anon ]
00007f69cc000000     132      12      12 rwx--   [ anon ]
00007f69cc021000   65404       0       0 -----   [ anon ]
00007f69d0000000     132      32      32 rwx--   [ anon ]
00007f69d0021000   65404       0       0 -----   [ anon ]
00007f69d4000000     132       8       8 rwx--   [ anon ]
00007f69d4021000   65404       0       0 -----   [ anon ]
00007f69d8000000   65320   64952   64952 rwx--   [ anon ]
00007f69dbfca000     216       0       0 -----   [ anon ]
00007f69dc085000    2124     560       0 r-xs- 000728.ldb
00007f69dcae3000    2088      40       0 r-xs- 000738.ldb
00007f69dccf2000    2104     200       0 r-xs- 000739.ldb
00007f69dcf00000    2124     732       0 r-xs- 000725.ldb
00007f69dd539000    2124     660       0 r-xs- 000721.ldb
00007f69dd755000    2088    1868       0 r-xs- 000425.ldb
00007f69dd95f000    2096      48       0 r-xs- 000703.ldb
00007f69ddb6b000    2096      48       0 r-xs- 000702.ldb
00007f69ddd7a000    2084    1884       0 r-xs- 000423.ldb
00007f69ddf83000    2124     108       0 r-xs- 000708.ldb
00007f69de196000       4       0       0 -----   [ anon ]
00007f69de197000    8192      12      12 rwx--   [ anon ]
00007f69de997000    2124     556       0 r-xs- 000722.ldb
00007f69debaa000    2124     608       0 r-xs- 000733.ldb
00007f69dedc7000    2084    1784       0 r-xs- 000426.ldb
00007f69df1d9000    2124     468       0 r-xs- 000726.ldb
00007f69df5ec000    2124     552       0 r-xs- 000730.ldb
00007f69df7ff000       4       0       0 -----   [ anon ]
00007f69df800000    8192    2048    2048 rwx--   [ anon ]
00007f69e0000000    3972     136     136 rwx--   [ anon ]
00007f69e03e1000   61564       0       0 -----   [ anon ]
00007f69e4042000       4       0       0 -----   [ anon ]
00007f69e4043000    8192      20      20 rwx--   [ anon ]
00007f69e4843000       4       0       0 -----   [ anon ]
00007f69e4844000    8192      40      40 rwx--   [ anon ]
00007f69e5044000       4       0       0 -----   [ anon ]
00007f69e5045000    8192      20      20 rwx--   [ anon ]
00007f69e5845000       4       0       0 -----   [ anon ]
00007f69e5846000    8192      20      20 rwx--   [ anon ]
00007f69e6046000       4       0       0 -----   [ anon ]
00007f69e6047000    8192      84      84 rwx--   [ anon ]
00007f69e6847000       4       0       0 -----   [ anon ]
00007f69e6848000    8192      20      20 rwx--   [ anon ]
00007f69e7048000       4       0       0 -----   [ anon ]
00007f69e7049000    8192      20      20 rwx--   [ anon ]
00007f69e7849000       4       0       0 -----   [ anon ]
00007f69e784a000    8192      24      24 rwx--   [ anon ]
00007f69e804a000       4       0       0 -----   [ anon ]
00007f69e804b000    8192       8       8 rwx--   [ anon ]
00007f69e8a4b000    2124     588       0 r-xs- 000734.ldb
00007f69e8c5e000    2084    1820       0 r-xs- 000434.ldb
00007f69e9070000    2084    1868       0 r-xs- 000436.ldb
00007f69e9279000    2088    1852       0 r-xs- 000431.ldb
00007f69e967b000    2124     544       0 r-xs- 000715.ldb
00007f69e988e000    2084    1872       0 r-xs- 000433.ldb
00007f69e9a97000    2124      84       0 r-xs- 000727.ldb
00007f69e9caa000    2084    1812       0 r-xs- 000424.ldb
00007f69e9ff0000    2120      80       0 r-xs- 000719.ldb
00007f69ea202000    2124     480       0 r-xs- 000740.ldb
00007f69ea415000    2084    1784       0 r-xs- 000422.ldb
00007f69ea61e000    2084    1824       0 r-xs- 000421.ldb
00007f69eaa31000    2088    2088       0 r-xs- 000289.ldb
00007f69eac3b000    2088    2088       0 r-xs- 000273.ldb
00007f69eae45000    2088    2088       0 r-xs- 000271.ldb
00007f69eb249000    2124     684       0 r-xs- 000724.ldb
00007f69eb45d000    2120     552       0 r-xs- 000731.ldb
00007f69eb66f000    4100    4100    4100 rwx--   [ anon ]
00007f69eba70000    2092    2092       0 r-xs- 000268.ldb
00007f69ebc7b000    2088    2088       0 r-xs- 000267.ldb
00007f69ebef2000    2084    1820       0 r-xs- 000438.ldb
00007f69ec0fb000    2084    1916       0 r-xs- 000439.ldb
00007f69ec3c5000    2124     644       0 r-xs- 000710.ldb
00007f69ec695000    2084    1848       0 r-xs- 000430.ldb
00007f69ec8a8000    2084    1796       0 r-xs- 000435.ldb
00007f69ecab1000    2124      92       0 r-xs- 000736.ldb
00007f69eccc4000    2088    2088       0 r-xs- 000405.ldb
00007f69ecf25000    2124     536       0 r-xs- 000717.ldb
00007f69ed138000    1704      36       0 r-xs- 000701.ldb
00007f69ed2e2000    2088    2088       0 r-xs- 000402.ldb
00007f69ed544000     580     496       0 r-xs- 000437.ldb
00007f69ed658000    2124     600       0 r-xs- 000735.ldb
00007f69ed86b000    2084    1852       0 r-xs- 000432.ldb
00007f69eda74000    2124     568       0 r-xs- 000732.ldb
00007f69edc87000     424     108       0 r-xs- 000729.ldb
00007f69edd92000    2072    1840       0 r-xs- 000440.ldb
00007f69edf98000   11700   11700   11700 rwx--   [ anon ]
00007f69eeb05000       4       0       0 -----   [ anon ]
00007f69eeb06000    8192      16      16 rwx--   [ anon ]
00007f69ef306000       4       0       0 -----   [ anon ]
00007f69ef307000    8192       0       0 rwx--   [ anon ]
00007f69efb07000       4       0       0 -----   [ anon ]
00007f69efb08000    8192       0       0 rwx--   [ anon ]
00007f69f0308000       4       0       0 -----   [ anon ]
00007f69f0309000    8192      16      16 rwx--   [ anon ]
00007f69f0b09000       4       0       0 -----   [ anon ]
00007f69f0b0a000    8192       8       8 rwx--   [ anon ]
00007f69f130a000    2852       0       0 r-x-- locale-archive
00007f69f15d3000    1772     520       0 r-x-- libc-2.19.so
00007f69f178e000    2044       0       0 ----- libc-2.19.so
00007f69f198d000      16      12      12 r-x-- libc-2.19.so
00007f69f1991000       8       8       8 rwx-- libc-2.19.so
00007f69f1993000      20      16      16 rwx--   [ anon ]
00007f69f1998000     100      72       0 r-x-- libpthread-2.19.so
00007f69f19b1000    2044       0       0 ----- libpthread-2.19.so
00007f69f1bb0000       4       4       4 r-x-- libpthread-2.19.so
00007f69f1bb1000       4       4       4 rwx-- libpthread-2.19.so
00007f69f1bb2000      16       4       4 rwx--   [ anon ]
00007f69f1bb6000      88      48       0 r-x-- libgcc_s.so.1
00007f69f1bcc000    2044       0       0 ----- libgcc_s.so.1
00007f69f1dcb000       4       4       4 rwx-- libgcc_s.so.1
00007f69f1dcc000    1044      88       0 r-x-- libm-2.19.so
00007f69f1ed1000    2044       0       0 ----- libm-2.19.so
00007f69f20d0000       4       4       4 r-x-- libm-2.19.so
00007f69f20d1000       4       4       4 rwx-- libm-2.19.so
00007f69f20d2000     920     384       0 r-x-- libstdc++.so.6.0.19
00007f69f21b8000    2044       0       0 ----- libstdc++.so.6.0.19
00007f69f23b7000      32      16      16 r-x-- libstdc++.so.6.0.19
00007f69f23bf000       8       8       8 rwx-- libstdc++.so.6.0.19
00007f69f23c1000      84      16      16 rwx--   [ anon ]
00007f69f23d6000      12       4       0 r-x-- libanl-2.19.so
00007f69f23d9000    2044       0       0 ----- libanl-2.19.so
00007f69f25d8000       4       4       4 r-x-- libanl-2.19.so
00007f69f25d9000       4       0       0 rwx-- libanl-2.19.so
00007f69f25da000      28       4       0 r-x-- librt-2.19.so
00007f69f25e1000    2044       0       0 ----- librt-2.19.so
00007f69f27e0000       4       4       4 r-x-- librt-2.19.so
00007f69f27e1000       4       0       0 rwx-- librt-2.19.so
00007f69f27e2000     140      40       0 r-x-- ld-2.19.so
00007f69f2819000     524     156       0 r-xs- 000737.ldb
00007f69f289c000       8       8       8 rwx--   [ anon ]
00007f69f28a1000      88      12       0 r-xs- 000742.ldb
00007f69f28bd000       4       4       0 r-xs- 000022.ldb
00007f69f28ed000    1044    1044    1044 rwx--   [ anon ]
00007f69f29f2000      28      12      12 rwx--   [ anon ]
00007f69f29f9000       4       0       0 rwx--   [ anon ]
00007f69f29fa000       4       0       0 rwx--   [ anon ]
00007f69f29fb000      28       0       0 r-xs- gconv-modules.cache
00007f69f2a02000       8       4       4 rwx--   [ anon ]
00007f69f2a04000       4       4       4 r-x-- ld-2.19.so
00007f69f2a05000       4       4       4 rwx-- ld-2.19.so
00007f69f2a06000       4       4       4 rwx--   [ anon ]
00007f69f2a07000    8972    3908       0 r-x-- dashd
00007f69f32cb000     168      76      76 r-x-- dashd
00007f69f32f5000      60      20      20 rwx-- dashd
00007f69f3304000     356     348     348 rwx--   [ anon ]
00007f69f452b000     272     156     156 rwx--   [ anon ]
00007f69f456f000       4       4       4 rwx--   [ anon ]
00007f69f4570000     128     120     120 rwx--   [ anon ]
00007f69f4590000    9672    9636    9636 rwx--   [ anon ]
00007f69f4f02000       4       4       4 rwx--   [ anon ]
00007f69f4f03000      44      44      44 rwx--   [ anon ]
00007f69f4f0e000       4       4       4 rwx--   [ anon ]
00007f69f4f0f000   65680   65680   65680 rwx--   [ anon ]
00007f69f8f33000      16      16      16 rwx--   [ anon ]
00007f69f8f37000     236     236     236 rwx--   [ anon ]
00007f69f8f72000       4       4       4 rwx--   [ anon ]
00007f69f8f73000      28      28      28 rwx--   [ anon ]
00007f69f8f7a000       4       4       4 rwx--   [ anon ]
00007f69f8f7b000      20      20      20 rwx--   [ anon ]
00007f69f8f80000       4       4       4 rwx--   [ anon ]
00007f69f8f81000    1116    1116    1116 rwx--   [ anon ]
00007f69f9098000       4       4       4 rwx--   [ anon ]
00007f69f9099000      48      48      48 rwx--   [ anon ]
00007f69f90a5000       8       8       8 rwx--   [ anon ]
00007f69f90a7000    3916    3916    3916 rwx--   [ anon ]
00007f69f947a000       4       4       4 rwx--   [ anon ]
00007f69f947b000    9724    9724    9724 rwx--   [ anon ]
00007f69f9dfa000       8       8       8 rwx--   [ anon ]
00007f69f9dfc000      40       4       4 rwx--   [ anon ]
00007fff2a866000     308      16      16 rwx--   [ stack ]
00007fff2a991000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1426136  625224  559000
schinzelh commented 9 years ago

I had a buffer overflow recently

*** buffer overflow detected ***: dashd terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x73e0f)[0x7fce0fb64e0f] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7fce0fbfc71c] /lib/x86_64-linux-gnu/libc.so.6(+0x10a5e0)[0x7fce0fbfb5e0] dashd(+0x272c15)[0x7fce11198c15] dashd(+0x139a4c)[0x7fce1105fa4c] dashd(+0x14652a)[0x7fce1106c52a] dashd(+0x4235b7)[0x7fce113495b7] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7fce0febf182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fce0fbebefd]

eduffield222 commented 9 years ago

I'm looking into this. I'll let you know when I figure it out

minersday commented 9 years ago

followup, nodes mem usage 1: node1-memusage2 2: node2-memusage2

schinzelh commented 9 years ago

Yep, same here. Nevertheless the CPU load from startup to startup+5min is quite high - seems like there is still a lot of 'nwm' message flooding during this time.