search

ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.16k stars 19.96k forks source link

Geth 1.7.0: fatal error: runtime: out of memory #15157

Closed vogelito closed 6 years ago

vogelito commented 6 years ago

System information

Geth version:

geth version
Geth
Version: 1.7.0-stable
Git Commit: 6c6c7b2af3efdad4d2f64f70f3a724af434bbcd2
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.7
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

OS & Version: Ubuntu 14.04.5 LTS

Expected behaviour

Geth 1.6.6 never used as much memory

Actual behaviour

Geth 1.7.0 eats away all the memory and dies

Steps to reproduce the behaviour

This has been happening when doing a fresh sync on 1.7.0 in a box with 8GB RAM

Backtrace

Full backtrace is more than 2,000 lines, showing the first few ones:

2017-09-18 02:38:53.726471500 fatal error: runtime: out of memory
2017-09-18 02:38:53.739815500 
2017-09-18 02:38:53.739815500 runtime stack:
2017-09-18 02:38:53.748015500 runtime.throw(0xf3747c, 0x16)
2017-09-18 02:38:53.749998500   /usr/local/go/src/runtime/panic.go:566 +0x95
2017-09-18 02:38:53.749999500 runtime.sysMap(0xc5d8b30000, 0x100000, 0x11c0900, 0x1859dd8)
2017-09-18 02:38:53.750533500   /usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
2017-09-18 02:38:53.750535500 runtime.(*mheap).sysAlloc(0x18403a0, 0x100000, 0x18403b8)
2017-09-18 02:38:53.750603500   /usr/local/go/src/runtime/malloc.go:407 +0x37a
2017-09-18 02:38:53.750605500 runtime.(*mheap).grow(0x18403a0, 0x1, 0x0)
2017-09-18 02:38:53.750606500   /usr/local/go/src/runtime/mheap.go:726 +0x62
2017-09-18 02:38:53.750607500 runtime.(*mheap).allocSpanLocked(0x18403a0, 0x1, 0x7ff300000001)
2017-09-18 02:38:53.750609500   /usr/local/go/src/runtime/mheap.go:630 +0x4f2
2017-09-18 02:38:53.750610500 runtime.(*mheap).alloc_m(0x18403a0, 0x1, 0x7f0000000013, 0x7ff32e694720)
2017-09-18 02:38:53.750685500   /usr/local/go/src/runtime/mheap.go:515 +0xe0
2017-09-18 02:38:53.751627500 runtime.(*mheap).alloc.func1()
2017-09-18 02:38:53.751627500   /usr/local/go/src/runtime/mheap.go:579 +0x4b
2017-09-18 02:38:53.751628500 runtime.systemstack(0x7ff35375bd48)
2017-09-18 02:38:53.751629500   /usr/local/go/src/runtime/asm_amd64.s:314 +0xab
2017-09-18 02:38:53.751629500 runtime.(*mheap).alloc(0x18403a0, 0x1, 0x10000000013, 0x42c329)
2017-09-18 02:38:53.751630500   /usr/local/go/src/runtime/mheap.go:580 +0x73
2017-09-18 02:38:53.751630500 runtime.(*mcentral).grow(0x1841ec0, 0x0)
2017-09-18 02:38:53.751678500   /usr/local/go/src/runtime/mcentral.go:210 +0x94
2017-09-18 02:38:53.751679500 runtime.(*mcentral).cacheSpan(0x1841ec0, 0xc4a2266c00)
2017-09-18 02:38:53.751679500   /usr/local/go/src/runtime/mcentral.go:91 +0xfa
2017-09-18 02:38:53.751680500 runtime.(*mcache).refill(0x7ff3878794b0, 0x13, 0xc517f3e280)
2017-09-18 02:38:53.751681500   /usr/local/go/src/runtime/mcache.go:121 +0xae
2017-09-18 02:38:53.751681500 runtime.(*mcache).nextFree.func1()
2017-09-18 02:38:53.751708500   /usr/local/go/src/runtime/malloc.go:505 +0x33
2017-09-18 02:38:53.751709500 runtime.systemstack(0xc420012000)
2017-09-18 02:38:53.751709500   /usr/local/go/src/runtime/asm_amd64.s:298 +0x79
2017-09-18 02:38:53.751710500 runtime.mstart()
2017-09-18 02:38:53.751710500   /usr/local/go/src/runtime/proc.go:1079

The black vertical line shows the moment I shut down Geth 1.6.6 and started a fresh sync for 1.7.0.

This is CPU:

screen shot 2017-09-17 at 10 28 29 pm

This is memory:

screen shot 2017-09-17 at 10 28 17 pm

The graphs don't have full resolution, but the dip in memory consumption is geth restarting after the box becomes unusable. This never happened on the same box with prior versions of geth.

francofs commented 5 years ago

cors domain set to localhost to make sure it was not about brute force attacks.

Cors domain settings won't protect you from bruteforce-attacks from the internet, only protect you from pages you visit with your own browser from being able to read data from the api (they can still write/POST though)

You're right, that was an oversight, the thing is that disable the db rpc solved the issue. Not sure why an attacker would use that particular rpc api, so I suspect the out of memory problem is not related to an attack. I still have to test by closing the port on the firewall.

chrisciszak commented 5 years ago

removing db from RPC API didnt resolve the problem for us.

nastasache commented 5 years ago

I think is about 32bit vs 64bit. I found same geth.exe version on ProgramFiles and works. Used before from ProgramFiles(x86).On Mar 28, 2018 5:29 PM, alvin notifications@github.com wrote:I got this problem too. Geth:1.8.2-stable 8G memory it could be solved by creating a swap file,but is there some way to limit the memory Geth used ? changing --cache seems nothing happend

—You are receiving this because you commented.Reply to this email directly, view it on GitHub, or mute the thread.