fatal error: runtime: out of memory

[kimmobrunfeldt]

After 9h 30min running, my planet import process failed to out of memory.

I know I'm using a quite limited server in DigitalOcean: 1GB Memory, 1 Core Processor, 30GB SSD Disk, plus additional 350GB SSD which is used for imposm3 cache directory and postgres data directory.

Is there any way I could get the initial import working with my limited resources e.g. by increasing swap or limiting imposm3 memory usage? If not, what are the minimum RAM and disk space requirements?

I'm currently importing the planet with the following command, using imposm3-0.2.0dev-20160615-d495ca4-linux-x86-64 prebuilt binary.

/imposm3 import -mapping mapping.json -read /mnt/volume1/data/planet-latest.osm.pbf -write -connection postgis://osm:osm@localhost/osm -optimize -deployproduction -overwritecache -cachedir /mnt/volume1/data/.imposmcache

Partial stack trace below:

[Aug 29 08:09:20] [INFO] [9h28m0s] C:       0/s ( 0.0%) N:       0/s ( 0.0%) W:       0/s ( 0.0%) R:     60/s (89.8%)
fatal error: runtime: out of memory   0/s       0/s ( 0.0%) N:       0/s       0/s ( 0.0%) W:       0/s       0/s ( 0.0%) R:     60/s      0/s (89.8%)

runtime stack:
runtime.throw(0xab54c0, 0x16)
        /root/imposm/go/src/runtime/panic.go:530 +0x90
runtime.sysMap(0xc836330000, 0xba0000, 0x0, 0xf4d638)
        /root/imposm/go/src/runtime/mem_linux.go:206 +0x9b
runtime.(*mheap).sysAlloc(0xf33640, 0xba0000, 0x7f231fffebf8)
        /root/imposm/go/src/runtime/malloc.go:429 +0x191
runtime.(*mheap).grow(0xf33640, 0x5d0, 0x0)
        /root/imposm/go/src/runtime/mheap.go:651 +0x63
runtime.(*mheap).allocSpanLocked(0xf33640, 0x5cb, 0x8a03918)
        /root/imposm/go/src/runtime/mheap.go:553 +0x4f6
runtime.(*mheap).alloc_m(0xf33640, 0x5cb, 0x100000000, 0x7f2307544208)
        /root/imposm/go/src/runtime/mheap.go:437 +0x119
runtime.(*mheap).alloc.func1()
        /root/imposm/go/src/runtime/mheap.go:502 +0x41
runtime.systemstack(0x7f231fffede8)
        /root/imposm/go/src/runtime/asm_amd64.s:307 +0xab
runtime.(*mheap).alloc(0xf33640, 0x5cb, 0x10100000000, 0x7f2307544270)
        /root/imposm/go/src/runtime/mheap.go:503 +0x63
runtime.largeAlloc(0xb96000, 0x0, 0xc82c330000)
        /root/imposm/go/src/runtime/malloc.go:766 +0xb3
runtime.mallocgc.func3()
        /root/imposm/go/src/runtime/malloc.go:664 +0x33
runtime.systemstack(0xc82001c000)
        /root/imposm/go/src/runtime/asm_amd64.s:291 +0x79
runtime.mstart()
        /root/imposm/go/src/runtime/proc.go:1048

goroutine 50 [running]:
runtime.systemstack_switch()
        /root/imposm/go/src/runtime/asm_amd64.s:245 fp=0xc8200491c0 sp=0xc8200491b8
runtime.mallocgc(0xb96000, 0x9a9620, 0x0, 0x2e6000)
        /root/imposm/go/src/runtime/malloc.go:665 +0x9eb fp=0xc820049298 sp=0xc8200491c0
runtime.newarray(0x9a9620, 0x5cb00, 0x20)
        /root/imposm/go/src/runtime/malloc.go:798 +0xc9 fp=0xc8200492d8 sp=0xc820049298
runtime.growslice(0x894e80, 0xc8203950e0, 0x4, 0x5, 0x5caef, 0x0, 0x0, 0x0)
        /root/imposm/go/src/runtime/slice.go:100 +0x2c1 fp=0xc820049348 sp=0xc8200492d8
runtime.growslice_n(0x894e80, 0xc8203950e0, 0x4, 0x5, 0x5caea, 0x0, 0x0, 0x0)
        /root/imposm/go/src/runtime/slice.go:44 +0xc7 fp=0xc8200493a0 sp=0xc820049348
github.com/omniscale/imposm3/geom.mergeRings(0xc826ca0000, 0x11f3, 0x1400, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/geom/ring.go:101 +0x1d32 fp=0xc820049770 sp=0xc8200493a0
github.com/omniscale/imposm3/geom.buildRings(0xc8209c3410, 0x3fb999999999999a, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/geom/multipolygon.go:95 +0x4d5 fp=0xc8200499a0 sp=0xc820049770
github.com/omniscale/imposm3/geom.PrepareRelation(0xc8209c3410, 0xf11, 0x3fb999999999999a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/geom/multipolygon.go:21 +0x73 fp=0xc820049a20 sp=0xc8200499a0
github.com/omniscale/imposm3/writer.handleMultiPolygon(0xc82081d810, 0xc8209c3410, 0xc822211c60, 0x100)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/writer/relations.go:139 +0x85 fp=0xc820049d88 sp=0xc820049a20
github.com/omniscale/imposm3/writer.(*RelationWriter).loop(0xc82081d810)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/writer/relations.go:112 +0x461 fp=0xc820049f98 sp=0xc820049d88
runtime.goexit()
        /root/imposm/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820049fa0 sp=0xc820049f98
created by github.com/omniscale/imposm3/writer.(*OsmElemWriter).Start
        /root/imposm/gopath/src/github.com/omniscale/imposm3/writer/writer.go:55 +0xa3

goroutine 1 [semacquire, 568 minutes]:
sync.runtime_Semacquire(0xc822211c5c)
        /root/imposm/go/src/runtime/sema.go:47 +0x26
sync.(*WaitGroup).Wait(0xc822211c50)
        /root/imposm/go/src/sync/waitgroup.go:127 +0xb4
github.com/omniscale/imposm3/writer.(*OsmElemWriter).Wait(0xc82081d810)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/writer/writer.go:64 +0x25
github.com/omniscale/imposm3/import_.Import()
        /root/imposm/gopath/src/github.com/omniscale/imposm3/import_/import.go:191 +0x1ba8
github.com/omniscale/imposm3/cmd.Main(0xb4bea8)
        /root/imposm/gopath/src/github.com/omniscale/imposm3/cmd/main.go:46 +0x2d8
main.main()
        /root/imposm/gopath/src/github.com/omniscale/imposm3/imposm3.go:8 +0x23

... [CONTINUES]

[olt]

1GB is way to small for processing a planet file. You can reduce the memory usage of Imposm by creating a custom cache config [1], but it should still take ages. Importing OSM data is not a streaming operation. It needs to access the coordinate cache when buildings ways and relations and this cache is around 25 GB for a planet. Without any caching in memory, you will hit the disk for nearly every single way and relation.

[1] See https://github.com/omniscale/imposm3/blob/master/cache/config.go Point IMPOSM_CACHE_CONFIG environment to a custom config file. Use defaultConfig as a guidance.

[kimmobrunfeldt]

Ok thanks for the information. To me it seems that the default cache limitations are quite small, or does "BunchCacheCapacity": 8096 mean that imposm can use 8GB memory?

Do you think the planet import would work with 16GB RAM?

edit: I just found this benchmark section: https://github.com/omniscale/imposm3/wiki/Benchmarks, so at least someone has successfully imported the planet data with 16GB RAM.

[olt]

It's the memory the underlying LevelDB uses for internal caching for each of the different LevelDB (Imposm cache) files. The LevelDB files them self can become much larger and the OS will use all your "free" memory to cache these files, so that it doesn't have to hit the disk for each read operation.

The WriteBufferSizeM can be allocated twice by LevelDB during write operations. This is why the actual memory consumption of Imposm is larger then the sum of the SizeM options.

BunchCacheCapacity is the size of how many blocks of compressed coordinates are stored.

For full speed you will need 64GB of RAM nowadays. It should theoretically work with 2GB, but I guess it would need a couple of days, if not weeks on such a machine.