ethereum / go-ethereum

Go implementation of the Ethereum protocol
https://geth.ethereum.org
GNU Lesser General Public License v3.0
47.07k stars 19.93k forks source link

Geth 1.7.0: fatal error: runtime: out of memory #15157

Closed vogelito closed 6 years ago

vogelito commented 6 years ago

System information

Geth version:

geth version
Geth
Version: 1.7.0-stable
Git Commit: 6c6c7b2af3efdad4d2f64f70f3a724af434bbcd2
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.7
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

OS & Version: Ubuntu 14.04.5 LTS

Expected behaviour

Geth 1.6.6 never used as much memory

Actual behaviour

Geth 1.7.0 eats away all the memory and dies

Steps to reproduce the behaviour

This has been happening when doing a fresh sync on 1.7.0 in a box with 8GB RAM

Backtrace

Full backtrace is more than 2,000 lines, showing the first few ones:

2017-09-18 02:38:53.726471500 fatal error: runtime: out of memory
2017-09-18 02:38:53.739815500 
2017-09-18 02:38:53.739815500 runtime stack:
2017-09-18 02:38:53.748015500 runtime.throw(0xf3747c, 0x16)
2017-09-18 02:38:53.749998500   /usr/local/go/src/runtime/panic.go:566 +0x95
2017-09-18 02:38:53.749999500 runtime.sysMap(0xc5d8b30000, 0x100000, 0x11c0900, 0x1859dd8)
2017-09-18 02:38:53.750533500   /usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
2017-09-18 02:38:53.750535500 runtime.(*mheap).sysAlloc(0x18403a0, 0x100000, 0x18403b8)
2017-09-18 02:38:53.750603500   /usr/local/go/src/runtime/malloc.go:407 +0x37a
2017-09-18 02:38:53.750605500 runtime.(*mheap).grow(0x18403a0, 0x1, 0x0)
2017-09-18 02:38:53.750606500   /usr/local/go/src/runtime/mheap.go:726 +0x62
2017-09-18 02:38:53.750607500 runtime.(*mheap).allocSpanLocked(0x18403a0, 0x1, 0x7ff300000001)
2017-09-18 02:38:53.750609500   /usr/local/go/src/runtime/mheap.go:630 +0x4f2
2017-09-18 02:38:53.750610500 runtime.(*mheap).alloc_m(0x18403a0, 0x1, 0x7f0000000013, 0x7ff32e694720)
2017-09-18 02:38:53.750685500   /usr/local/go/src/runtime/mheap.go:515 +0xe0
2017-09-18 02:38:53.751627500 runtime.(*mheap).alloc.func1()
2017-09-18 02:38:53.751627500   /usr/local/go/src/runtime/mheap.go:579 +0x4b
2017-09-18 02:38:53.751628500 runtime.systemstack(0x7ff35375bd48)
2017-09-18 02:38:53.751629500   /usr/local/go/src/runtime/asm_amd64.s:314 +0xab
2017-09-18 02:38:53.751629500 runtime.(*mheap).alloc(0x18403a0, 0x1, 0x10000000013, 0x42c329)
2017-09-18 02:38:53.751630500   /usr/local/go/src/runtime/mheap.go:580 +0x73
2017-09-18 02:38:53.751630500 runtime.(*mcentral).grow(0x1841ec0, 0x0)
2017-09-18 02:38:53.751678500   /usr/local/go/src/runtime/mcentral.go:210 +0x94
2017-09-18 02:38:53.751679500 runtime.(*mcentral).cacheSpan(0x1841ec0, 0xc4a2266c00)
2017-09-18 02:38:53.751679500   /usr/local/go/src/runtime/mcentral.go:91 +0xfa
2017-09-18 02:38:53.751680500 runtime.(*mcache).refill(0x7ff3878794b0, 0x13, 0xc517f3e280)
2017-09-18 02:38:53.751681500   /usr/local/go/src/runtime/mcache.go:121 +0xae
2017-09-18 02:38:53.751681500 runtime.(*mcache).nextFree.func1()
2017-09-18 02:38:53.751708500   /usr/local/go/src/runtime/malloc.go:505 +0x33
2017-09-18 02:38:53.751709500 runtime.systemstack(0xc420012000)
2017-09-18 02:38:53.751709500   /usr/local/go/src/runtime/asm_amd64.s:298 +0x79
2017-09-18 02:38:53.751710500 runtime.mstart()
2017-09-18 02:38:53.751710500   /usr/local/go/src/runtime/proc.go:1079

The black vertical line shows the moment I shut down Geth 1.6.6 and started a fresh sync for 1.7.0.

This is CPU:

screen shot 2017-09-17 at 10 28 29 pm

This is memory:

screen shot 2017-09-17 at 10 28 17 pm

The graphs don't have full resolution, but the dip in memory consumption is geth restarting after the box becomes unusable. This never happened on the same box with prior versions of geth.

ThatOtherZach commented 6 years ago

Been having this issue for almost a few weeks now. I've tried setting the cache value and using lightkdf and yet when it gets to block ~3.7M it crashes. Using Supervisord does help keep the thing running, but it's still constantly crashing.

The only way I've gotten geth to run easily has been in light mode, which of course limits my ability to do anything. This really needs to be fixed.

Using DigitalOcean Ubuntu 16 x64, 2GB RAM, 40GB Disk (yes not ideal, but we need this for testing).

Inigovd commented 6 years ago

Exact same issue. Can't even get it to fully sync. Using DO Ubuntu 16.04 4GB RAM. Slowly eats up memory then crashes, same error message.

ThatOtherZach commented 6 years ago

@Inigovd Try using Supervisor. Mine eventually did sync, but only because that program kept kicking it along.

Also, try creating a memory swap file. That also helped me a little.

Inigovd commented 6 years ago

Already have swap enabled. Will try with supervisor... Does yours not crash anymore after syncing? I want to build a commercial application using geth for automated transactions, but if it keeps crashing, I'll look into other options...

ThatOtherZach commented 6 years ago

I just checked mine, its still running. I hear Parity might be better, but I've yet to try it myself (plan B if this thing causes more issues). Depending on your apps requirements and expected traffic, check out Infura's api.

etscrivner commented 6 years ago

Coinbase also seeing this issue, it's causing a lot of geth instance churn right now.

WhisperingChaos commented 6 years ago

OS: Windows 7 Ultimate

Experienced same behavior when first syncing a new wallet. I filed it as an issue in Mist repository (see #3214). This issue contains attached node logs with panic stack that includes references to the ethereum packages that called into the golang runtime so it might provide further insight into the problem's cause.

Also, after "kicking it" three times, the sync completed. However, running geth constantly for 3-4 days after the sync resulted in same memory depletion issue.

If you use the computer for tasks other than running geth, it affects their performance too, especially if the OS utilizes virtual memory pages, which most do. I've disabled support for virtual memory on my Windows machine, so it doesn't suffer from the long slow death due to page thrashing that others have reported as block processing slows to a snails pace.

Finally, many issues similar to this one appear in the Mist repository which leads me to believe its a pervasive one that's under reported due to differences in how individuals find similar posts that reflect the behavior they are experiencing.

Inigovd commented 6 years ago

I had to 'kick it' as well a few times, just keep rebooting until it was fully synced. After that, I didn't have any issues. Memory usages stable now for 7+ days.

Note that I am using swap files as described here https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04, doing this made sure the instance didn't crash

On linux, I have the following cronjob running to make sure geth keeps running: * * * * * /usr/bin/flock -n /tmp/fcj2.lockfile /usr/bin/geth --rpc

Flock makes sure only one instance runs. Make sure that geth is started by this cronjob. Should work well to keep it running so it fully syncs, and keep it running after as well.

naure commented 6 years ago

Or even better, use systemd to keep it running while logging everything.

Adjust the following with your username and paths and save it as /etc/systemd/system/geth.service:

[Unit]

[Service]
User=user
WorkingDirectory=/home/user/
Environment=
ExecStart=/home/user/geth/geth --rpc
Restart=always

[Install]
WantedBy=default.target

And run:

systemctl daemon-reload
systemctl enable geth
systemctl restart geth
stevenroose commented 6 years ago

Is this solved in current geth? I think I'm experiencing this in Quorum (i.e. geth 1.7.2).

prock-fife commented 6 years ago

I don't believe so. Here is a snippet of the output after I run geth(no options):

.
.
.
INFO [01-24|07:18:45] Imported new state entries               count=228 elapsed=0s        processed=228 pending=2920 retry=19 duplicate=0 unexpected=0
INFO [01-24|07:18:49] Upgrading chain index                    type=bloombits percentage=34
INFO [01-24|07:18:52] Imported new block receipts              count=56  elapsed=268.015ms bytes=3722571 number=4194015 hash=0f06dc…173063 ignored=0
INFO [01-24|07:18:57] Upgrading chain index                    type=bloombits percentage=35
INFO [01-24|07:18:58] Imported new block receipts              count=44  elapsed=275.016ms bytes=3712962 number=4194059 hash=713c3b…d6fe3f ignored=0
fatal error: out of memory

runtime stack:
runtime.throw(0xcd7cf3, 0xd)
        C:/go/src/runtime/panic.go:605 +0x7c
runtime.(*mcache).refill(0x3ed035c, 0x7fdcb333, 0x73a0786c)
        C:/go/src/runtime/mcache.go:125 +0xee
runtime.(*mcache).nextFree.func1()
        C:/go/src/runtime/malloc.go:557 +0x28
runtime.systemstack(0x148fa000)
        C:/go/src/runtime/asm_386.s:393 +0x5e
runtime.mstart()
        C:/go/src/runtime/proc.go:1125
.
.
.

Windows 10, 8GB RAM

geth version
Geth
Version: 1.7.2-stable
Git Commit: 1db4ecdc0b9e828ff65777fb466fc7c1d04e0de9
Architecture: 386
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9
Operating System: windows
GOPATH=
GOROOT=c:\go
abronte commented 6 years ago

Seeing this as well in the latest version.

Geth
Version: 1.7.3-stable
Git Commit: 4bb3c89d44e372e6a9ab85a8be0c9345265c763a
Architecture: amd64
Protocol Versions: [63 62]
Network Id: 1
Go Version: go1.9
Operating System: linux
GOPATH=
GOROOT=/usr/lib/go-1.9
Duiesel commented 6 years ago

Same for me! Ubuntu server 16.04 Geth/v1.7.3-stable-4bb3c89d/linux-amd64/go1.9

kot-begemot commented 6 years ago

Make sure you have swap file allocated. Geth seems to be memory hungry and it eats all the memory available on your machine

    $ sudo fallocate -l 4G /media/storage/swapfile
    $ sudo chmod 600 /media/storage/swapfile
    $ sudo mkswap /media/storage/swapfile
    $ sudo swapon /media/storage/swapfile
Duiesel commented 6 years ago

Enable swap on 8Gb RAM machine seems to resolve this problem for me. (+ 8Gb swap) @kot-begemot writes above about it.

sudo dd if=/dev/zero of=/path/SWAPFILE bs=1024 count=SWAP_FILE_SIZE_IN_Kb
sudo chmod 600 /path/SWAPFILE
sudo mkswap /path/SWAPFILE
sudo swapon /path/SWAPFILE
naure commented 6 years ago

@Duiesel I still had some crashes with 16GB, no swap, nothing else running.

Duiesel commented 6 years ago

@Duiesel Try to create 16 GB swap file like I explain above and don't use --cache option.

kot-begemot commented 6 years ago

@naure create swap file and try again

poleguy commented 6 years ago

Still seeing it here on ubuntu 14.04.3 Geth Version: 1.7.3-stable Git Commit: 4bb3c89d44e372e6a9ab85a8be0c9345265c763a Architecture: amd64 Protocol Versions: [63 62] Network Id: 1 Go Version: go1.9 Operating System: linux GOPATH= GOROOT=/usr/lib/go-1.9

Distributor ID: Ubuntu Description: Ubuntu 14.04.3 LTS Release: 14.04 Codename: trusty

Running in VM.

I'm a developer and willing to try to help out. Can anyone suggest how best I could help debug/fix this?

kot-begemot commented 6 years ago

@poleguy How much memory do you have available there? Have you created swap file?

mariam-crissi commented 6 years ago

@stevenroose I am also facing this issue in Quorum.How did you resolve the out of memory issue in Quorum?

markusbkoch commented 6 years ago

Here's a more detailed explanation on setting up the swap file https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-16-04

stevenroose commented 6 years ago

@mariam-crissi We solved it by creating a big swap file.

poleguy commented 6 years ago

I am running in a VM, and it did have too little memory. I just tried with more memory, and it doesn't crash immediately now. It would be nice if geth the would check available memory and issue a warning below some level.

It does seem memory usage is excessive, and everyone would benefit if we could identify and fix leaks and inefficiencies.

I imagine many new users like me are giving up due to the slow seemingly buggy behavior. I'm not impressed.

If you have to throw top of the line hardware at it, it seems ethereum needs to rethink some things. That shouldn't be necessary for a network aspiring to be a distributed computing platform.

Growing pains I hope.

On Feb 4, 2018 6:13 PM, "Max Zet" notifications@github.com wrote:

@poleguy https://github.com/poleguy How much memory do you have available there? Have you created swap file?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ethereum/go-ethereum/issues/15157#issuecomment-362952460, or mute the thread https://github.com/notifications/unsubscribe-auth/AMzQf3AbLC66j6-_s47KFpG7FWqbbfpOks5tRkeygaJpZM4PaZ0X .

Lyghtning commented 6 years ago

I'm receiving the same error, i've been trying to solo mine for seems like 4 days now, and i end up giving up and going to nanopool. I really just want to solo mine. So I finally figured out Geth and Ethminer need to run at the same time.

However, seems i cannot sync, I'm running 20GB ram on my machine, and an AMD Radeon 570

When i run geth --rpc --rpcaddr "localhost" --rpcport 8545 console

I get to the point it starts block sync, and i watched videos that said wait till it said Start Mining to launch the ethminer.exe -G.

however after a few minutes it just spams a ton of crap, first few lines are Out Of memory. so i decided to launch and watch my resource monitor... sure enough, i'm blasting through all 20GB?

Lyghtning commented 6 years ago

So it runs a while at about 3GB which is cool and fine.. then all the sudden it just climbs really fast.... wondering how to fix this

Lyghtning commented 6 years ago

Going to apologize, so i noticed above some are using a much newer version of geth, than i was... Really wish this was easier to research/follow for setting up solo mining....

jjacquet commented 6 years ago

Creating a giant swap file is the key. Thanks

Lyghtning commented 6 years ago

Got it all figured out, was using a horribly old version. Updated to newest release, took 2 days but fully synced and now running

nastasache commented 6 years ago

The same to me. Tried with geth 1.8.2 on Win 10 64bit, 64GB RAM using a 128 VM paging file.

C:\Program Files (x86)\Geth>geth --fast --cache=1024

INFO [03-19|21:48:46] Maximum peer count ETH=25 LES=0 total=25 INFO [03-19|21:48:46] Starting peer-to-peer node instance=Geth/v1.8.2-stable-b8b9f7f4/windows-386/go1.9.2 INFO [03-19|21:48:46] Allocated cache and file handles database=C:\Users\user\AppData\Roaming\Ethereum\geth\chaindata cache=768 handles=1024 INFO [03-19|21:48:49] Initialised chain configuration config="{ChainID: 1 Homestead: 1150000 DAO: 1920000 DAOSupport: true EIP150: 2463000 EIP155: 2675000 EIP158: 2675000 Byzantium: 4370000 Constantinople: Engine: ethash}" INFO [03-19|21:48:49] Disk storage enabled for ethash caches dir=C:\Users\user\AppData\Roaming\Ethereum\geth\ethash count=3 INFO [03-19|21:48:49] Disk storage enabled for ethash DAGs dir=C:\Users\user\AppData\Ethash count=2 INFO [03-19|21:48:49] Initialising Ethereum protocol versions="[63 62]" network=1 INFO [03-19|21:48:49] Loaded most recent local header number=360984 hash=649f56…eb77ca td=1774293430119158975 INFO [03-19|21:48:49] Loaded most recent local full block number=0 hash=d4e567…cb8fa3 td=17179869184 INFO [03-19|21:48:49] Loaded most recent local fast block number=354070 hash=0c64d3…dddafb td=1731890002872788089 INFO [03-19|21:48:49] Upgrading chain index type=bloombits percentage=0 INFO [03-19|21:48:49] Loaded local transaction journal transactions=0 dropped=0 INFO [03-19|21:48:49] Regenerated local transaction journal transactions=0 accounts=0 INFO [03-19|21:48:49] Starting P2P networking INFO [03-19|21:48:51] UDP listener up self=enode://604ba23cf3...5a8323ce@[::]:30303
INFO [03-19|21:48:51] RLPx listener up self=enode://604ba23cf3...5a8323ce@[::]:30303 INFO [03-19|21:48:51] IPC endpoint opened url=\\.\pipe\geth.ipc INFO [03-19|21:48:57] Upgrading chain index type=bloombits percentage=35 INFO [03-19|21:49:05] Upgrading chain index type=bloombits percentage=70 INFO [03-19|21:49:09] Block synchronisation started INFO [03-19|21:49:10] Imported new state entries count=241 elapsed=0s processed=642713 pending=3857 retry=0 duplicate=0 unexpected=0 INFO [03-19|21:49:10] Imported new block headers count=0 elapsed=4.998ms number=354262 hash=1adfeb…8ffc20 ignored=192 .............. INFO [03-19|21:51:26] Imported new state entries count=1288 elapsed=2.981ms processed=807995 pending=10319 retry=0 duplicate=0 unexpected=0 INFO [03-19|21:51:26] Imported new state entries count=961 elapsed=2.005ms processed=808956 pending=10281 retry=0 duplicate=0 unexpected=0 INFO [03-19|21:51:27] Imported new block receipts count=2048 elapsed=45.154ms number=453734 hash=99c914…538196 size=1.70mB ignored=0 runtime: out of memory: cannot allocate 201326592-byte block (1605296128 in use) fatal error: out of memory

Sys mem usage: image

nastasache commented 6 years ago

@Lyghtning which versions you used (old, new)? Thx

Mike-McEwan commented 6 years ago

I have hit the same problem with geth 1.8.1 and 1.8.2 on Windows 10 with 8gb ram. I notice the above actions are all designed to increase available RAM, but it looks like a memory leakage was introduced by code change in an earlier release., because Windows Task Manager showed 90% RAM used when ~6Gb was taken by the running processes (that is, 75%). If so, that means increasing memory will not solve the problem, just delay the occurrence of it. Memory leakage occurs when the code requests additional memory, and then doesn't free it. This can happen when an application is written for one platform, then ported onto another, because there are a number of platforms which do automatic garbage collection, so don't need the application code to do an explicit action to free memory, but I'm guessing Windows doesn't do that. If so, it would need the developers of geth to have a look at the changes introduced into the first version where this problem was noticed, and see if there needs to be explicit freeing of dynamically allocated memory.. I'm new to all this, so don't know how to contact them - if someone here does, please pass these thoughts on. Also, for those who are new and trying to mine, but not succeeding, I found that when I wiped the files and started again with geth 1.8.1, it synchronised in about 3 days, started mining, and confirmed mining one block, before the memory error caused it to stop. When I restarted geth, it hit the memory error every time, which may be because the "fast" sync doesn't happen with an existing installation. I may not be investigating this much further, as I am looking into this because I would like to create my own cryptocurrency based on the Ethereum package, but for those of you who want to mine for Ether, and tripping over this problem, you may want to consider a different approach. I don't know if it will work, but if you are getting the memory error, it might be worth renaming your Ethereum directory to Ethereum-old, then re-installing everything and restarting geth and ethminer. geth (from 1.8 on) will rebuild from scratch using the fast method, so will take 2-3 days, and then mining will start. It is likely to crash a couple of days later with the memory problem, but may have done some mining by then. If so, you might be best to do the same again - rename, re-install and restart. I expect you will get asked to confirm that you want to replace existing files. If so, respond yes, because what you are doing is replacing existing files with the blank version, so geth will use the fast approach. If it all goes horribly wrong, or you're not happy with the new setup, you can re-instate the old by renaming Ethereum-old to Ethereum, replacing what's there. You can then restart geth , which will pick up from where the last update left off. Hope this helps some of you out there :)

Mike-McEwan commented 6 years ago

I forgot to mention that I searched for this problem and this thread came up on top - if there is a more current thread elsewhere please let me know

TUTUBIG commented 6 years ago

I got this problem too. Geth:1.8.2-stable 8G memory it could be solved by creating a swap file,but is there some way to limit the memory Geth used ? changing --cache seems nothing happend

nastasache commented 6 years ago

Hi there, I believe the problem is about 32bit vs 64bit geth version. I found I have geth.exe (1.8.2-stable) on two locations (have no idea why, maybe I installed twice): C:\Program Files (x86)\Geth>geth.exe (this one running out of memory) C:\Program Files\Geth>geth.exe (this one running well) I tested with 64GB RAM and 128GB VM. Getting out of memory probably mean the app can't address more than ~4GB RAM. That mean 32bit app.

Mike-McEwan commented 6 years ago

@nastasache - I think you're right about your problem, as 64Gb RAM should last a while, and there are both 32 and 64 bit downloads available. However I'm running 64 bit, as I allocated 5Gb to cache and it used it okay (and ran out sooner). I've stopped trying to mine now as there is a new limit that has kicked in - my graphics card has 2Gb and latest processing needs more than that, so it can't mine anyway.

Mike-McEwan commented 6 years ago

@TUTUBIG - I'm not aware of a solution - adding more RAM avoids the problem for a time.

zzd1990421 commented 6 years ago

amazon ecs with 4core 8GB RAM 2 restarts happen everyday can anybody help me?

fatal error: runtime: out of memory

runtime stack: runtime.throw(0xf2284f, 0x16) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/panic.go:616 +0x81 runtime.sysMap(0xc7cbdd0000, 0x10000000, 0x0, 0x1b16438) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mem_linux.go:216 +0x20a runtime.(mheap).sysAlloc(0x1afcae0, 0x10000000, 0x0) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/malloc.go:470 +0xd4 runtime.(mheap).grow(0x1afcae0, 0x8000, 0x0) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mheap.go:907 +0x60 runtime.(mheap).allocSpanLocked(0x1afcae0, 0x8000, 0x1b16448, 0xc42734f200) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mheap.go:820 +0x301 runtime.(mheap).alloc_m(0x1afcae0, 0x8000, 0x7f9a06b50101, 0x43913a) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mheap.go:686 +0x118 runtime.(mheap).alloc.func1() /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mheap.go:753 +0x4d runtime.(mheap).alloc(0x1afcae0, 0x8000, 0x7f9a0c010101, 0x7f9a06b5ae70) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/mheap.go:752 +0x8a runtime.largeAlloc(0x10000000, 0x7f9a06b50101, 0x46ab2e) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/malloc.go:826 +0x94 runtime.mallocgc.func1() /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/malloc.go:721 +0x46 runtime.systemstack(0x0) /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/asm_amd64.s:409 +0x79 runtime.mstart() /home/travis/.gimme/versions/go1.10.1.linux.amd64/src/runtime/proc.go:1175

stevenroose commented 6 years ago

@zzd1990421 give the thing an 8GiB swap file.

zzd1990421 commented 6 years ago

@stevenroose you can refer to #16728 It's a production evvironment.I need stable geth version. It seems I have to wait.

Mike-McEwan commented 6 years ago

Like most people on this thread I decided to upgrade memory. I went from 8Gb to 16Gb and when I run Geth I use -6144 as the storage limit, and it works. So with this 6Gb limit, Windows Task Manager shows it actually uses about 9Gb, but it grows very slowly after 9Gb (currently 9.9Gb on my system, which is doing an update after 2 weeks offline) - but it's still working! So - a memory upgrade is likely to avoid the problem for you. The data suggest there is a memory leakage in a very small area, and previous posts indicate it was introduced around the 1.6 release (I think) - it would be good if the authors of changes at that time could have a look at the code to see what might be causing this.

vogelito commented 6 years ago

I'm closing this issue as it relates to an old version of geth and it seems like 1.8.10 has addressed this (and many other) issues.

kwaazaar commented 6 years ago

I updated to 1.8.10 and the out of memory error is gone. However, I now get this: "fatal error: runtime: failed to commit pages", which seems just a different error message for a similar error.

Geth keeps crashing on my Windows Server 2016 (hosted on Azure) while running in fast-sync mode. But looking at the replies above, Linux has similar problems.

pdagrawal commented 6 years ago

I am facing the same error. Geth Version: 1.8.14-stable-316fc7ec Operating System: ubuntu 16.04 LTS AWS Instance: 2GB RAM Swap: 4GB

laurisbernhart commented 6 years ago

image

image

samaujs commented 6 years ago

Hi, I have encountered the same out of memory problem after upgrade with the following configurations : Geth Version: 1.8.15-stable Git Commit: 89451f7c382ad2185987ee369f16416f89c28a7d Architecture: amd64 Protocol Versions: [63 62] Network Id: 1 Go Version: go1.10 Operating System: linux GOPATH= GOROOT=/usr/lib/go-1.10 AWS running Ubuntu 16.04.5 LTS (GNU/Linux 4.14.20-041420-generic x86_64) ...

I have tried couple of times and realised that it always fail around the same spot : INFO [09-04|04:06:44.440] Initialising Ethereum protocol versions="[63 62]" network=4 WARN [09-04|04:06:44.448] Head state missing, repairing chain number=2867862 hash=1e6e59…0a36ff INFO [09-04|04:07:08.704] Rewound blockchain to past state number=2837951 hash=635371…0df25a INFO [09-04|04:07:08.715] Loaded most recent local header number=2867865 hash=6fb6b1…b8cde5 td=5382446 INFO [09-04|04:07:08.716] Loaded most recent local full block number=2837951 hash=635371…0df25a td=5338101 INFO [09-04|04:07:08.716] Loaded most recent local fast block number=2867865 hash=6fb6b1…b8cde5 td=5382446 INFO [09-04|04:07:08.716] Loaded local transaction journal transactions=0 dropped=0 INFO [09-04|04:07:08.716] Regenerated local transaction journal transactions=0 accounts=0 WARN [09-04|04:07:08.716] Blockchain not empty, fast sync disabled ... Please assist. Thank you.

0xhiroooo commented 5 years ago

same error,

Ubuntu 16 go1.11.1 geth 1.8.14-stable memory 16G swap 16G(with @kot-begemot 's code )

with command:

geth --datadir "/root/eth" --syncmode "full" --rpc --rpcaddr "0.0.0.0" --rpcapi eth,web3,personal,db,net --ws --wsaddr "0.0.0.0" console

still out of memory

@vogelito @Duiesel Have u fixed this problem finally?

holiman commented 5 years ago

Most memory-related problems seem to occur when this is in combination: --rpcaddr "0.0.0.0" --rpcapi eth,web3,personal. Usually because attackers on the internet try to bruteforce your wallet, which is memory intensive. Please check your logs, or attach to a ticket if you want some help analysing it

francofs commented 5 years ago

same error,

Ubuntu 16 go1.11.1 geth 1.8.14-stable memory 16G swap 16G(with @kot-begemot 's code )

with command:

geth --datadir "/root/eth" --syncmode "full" --rpc --rpcaddr "0.0.0.0" --rpcapi eth,web3,personal,db,net --ws --wsaddr "0.0.0.0" console

still out of memory

@vogelito @Duiesel Have u fixed this problem finally?

Can you try start geth without db RPC api? My memory issues were solved after doing that. Isolated it after having the cors domain set to localhost to make sure it was not about brute force attacks.

holiman commented 5 years ago

cors domain set to localhost to make sure it was not about brute force attacks.

Cors domain settings won't protect you from bruteforce-attacks from the internet, only protect you from pages you visit with your own browser from being able to read data from the api (they can still write/POST though)