search

pelias / pbf2json

An OpenStreetMap pbf parser which exports json, allows you to cherry-pick tags and handles denormalizing ways and relations. Available as a standalone binary and comes with a convenient npm wrapper.
https://pelias.io
MIT License
143 stars 36 forks source link

Fatal Error Using pbf2json on OSX #45

Closed tigerlily-he closed 7 years ago

tigerlily-he commented 7 years ago

Encountered error when running Open Street Map importer. Tried pbf2json version 4.3.0 and version in https://github.com/pelias/pbf2json/pull/43.

OSX Version: 10.12.5 (16F73)

Go is not installed on this machine. 

openstreetmap git:(production) npm start

> pelias-openstreetmap@0.0.0-semantic-release start /Users/lilyhe/openstreetmap
> node index.js

2017-06-06T19:35:20.275Z - info: [openstreetmap] Creating read stream for: /Users/lilyhe/Data/openstreetmap/new-york_new-york.osm.pbf
2017-06-06T19:35:23.100Z - info: [wof-pip-service:master] dependency worker loaded 38 features in 1.568 seconds
2017-06-06T19:35:23.460Z - info: [wof-pip-service:master] borough worker loaded 107 features in 1.922 seconds
2017-06-06T19:35:26.105Z - info: [wof-pip-service:master] macrocounty worker loaded 372 features in 4.574 seconds
[pbf2json]: fatal error: unexpected signal during runtime execution
[pbf2json]: fatal error: unexpected signal during runtime execution
[pbf2json]:
[pbf2json]: [signal
[pbf2json]: 0xb
[pbf2json]: code=
[pbf2json]: 0x1
[pbf2json]: addr=
[pbf2json]: 0xf0f5413dc6a
[pbf2json]: pc=
[pbf2json]: 0x1287b
[pbf2json]: ]
[pbf2json]:
[pbf2json]: runtime stack:
[pbf2json]: runtime.throw(0x2ec720,
[pbf2json]: 0x2a)
[pbf2json]:     /usr/local/go/src/runtime/panic.go:547 +0x90
[pbf2json]: runtime.sigpanic()
[pbf2json]:     /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
[pbf2json]: runtime.unlock(0x3f3ea0)
[pbf2json]:     /usr/local/go/src/runtime/lock_sema.go:107
[pbf2json]: +0x14b
[pbf2json]: runtime.(*mheap).alloc_m
[pbf2json]: (0x3f3ea0, 0x1
[pbf2json]: , 0x4, 0x1a385e8)
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:492 +0x314
[pbf2json]: runtime.(*mheap).alloc.func1()
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:502 +0x41
[pbf2json]: runtime.systemstack(0xc820041e58)
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:307 +0xab
[pbf2json]: runtime.(*mheap).alloc(0x3f3ea0, 0x1, 0x10000000004,
[pbf2json]: 0x1251f)
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:503 +0x63
[pbf2json]: runtime.(*mcentral).grow(0x3f5330, 0x0)
[pbf2json]:     /usr/local/go/src/runtime/mcentral.go:209
[pbf2json]: +0x93
[pbf2json]: runtime.(*mcentral).cacheSpan(0x3f5330, 0x1a385e8)
[pbf2json]:     /usr/local/go/src/runtime/mcentral.go:89 +0x47d
[pbf2json]: runtime.(*mcache).refill(0x4654b0, 0x4, 0x1a385e8)
[pbf2json]:     /usr/local/go/src/runtime/mcache.go:119 +0xcc
[pbf2json]: runtime.mallocgc.func2()
[pbf2json]:     /usr/local/go/src/runtime/malloc.go:642 +0x2b
[pbf2json]: runtime.systemstack(0xc820022000)
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:291 +0x79
[pbf2json]: runtime.mstart()
[pbf2json]:     /usr/local/go/src/runtime/proc.go:1051
[pbf2json]:
[pbf2json]: goroutine 7 [running]:
[pbf2json]: runtime.systemstack_switch(
[pbf2json]: )
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:245 fp=0xc820215818 sp=0xc820215810
[pbf2json]: runtime.mallocgc(0x30, 0x26a880, 0x0, 0xc81ceea71c)
[pbf2json]:     /usr/local/go/src/runtime/malloc.go:643 +0x869 fp=0xc8202158f0 sp=0xc820215818
[pbf2json]: runtime.newobject(0x26a880, 0xc8822b1c70)
[pbf2json]:     /usr/local/go/src/runtime/malloc.go:781 +0x42 fp=0xc820215918 sp=0xc8202158f0
[pbf2json]: runtime.makemap(0x1da480, 0x0, 0x0, 0x0, 0xc88229dfb0)
[pbf2json]:     /usr/local/go/src/runtime/hashmap.go:257 +0x381 fp=0xc820215978 sp=0xc820215918
[pbf2json]: github.com/qedus/osmpbf.(*tagUnpacker).next(0xc820215c28, 0xc8822b1c90)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode_tag.go:22 +0x4a fp=0xc820215a20 sp=0xc820215978
[pbf2json]: github.com/qedus/osmpbf.(*dataDecoder).parseDenseNodes(0xc820215ef8, 0xc8201f0900, 0xc8211f6480)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode_data.go
[pbf2json]: :91 +0x492 fp=0xc820215cf8 sp=0xc820215a20
[pbf2json]: github.com/qedus/osmpbf.(*dataDecoder).parsePrimitiveGroup(0xc820215ef8, 0xc8201f0900, 0xc8211f6400)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode_data.go:40 +0xb1 fp=0xc820215d70 sp=0xc820215cf8
[pbf2json]: github.com/qedus/osmpbf.(*dataDecoder).parsePrimitiveBlock(0xc820215ef8, 0xc8201f0900)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode_data.go:34 +0xa1 fp=0xc820215dd8 sp=0xc820215d70
[pbf2json]: github.com/qedus/osmpbf.(*dataDecoder).Decode(0xc820215ef8, 0xc82001a080, 0x0, 0x0, 0x0, 0x0, 0x0)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode_data.go:28 +0x235 fp=0xc820215e88 sp=0xc820215dd8
[pbf2json]: github.com/qedus/osmpbf.(*Decoder).Start.func1(0xc82001c300, 0xc82001c360)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:143 +0x130 fp=0xc820215f90 sp=0xc820215e88
[pbf2json]: runtime.goexit()
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820215f98 sp=0xc820215f90
[pbf2json]: created by
[pbf2json]: github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:151 +0x2fd
[pbf2json]:
[pbf2json]: goroutine 1 [chan receive]:
[pbf2json]: main.run(0xc820018320, 0xc8201f4180, 0x7fff5fbfe449, 0x3a, 0x7fff5fbfe444, 0x4, 0xc820014ea0, 0xc350)
[pbf2json]:     /home/peter/.go/src/github.com/pelias/pbf2json/pbf2json.go:83 +0xf7
[pbf2json]: main.main()
[pbf2json]:     /home/peter/.go/src/github.com/pelias/pbf2json/pbf2json.go:74 +0x43b
[pbf2json]:
[pbf2json]: goroutine 5 [running]:
[pbf2json]:     goroutine running on other thread; stack unavailable
[pbf2json]: created by github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:151 +0x2fd
[pbf2json]:
[pbf2json]: goroutine 6 [running]:
[pbf2json]:     goroutine running on other thread; stack unavailable
[pbf2json]: created by github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:151 +0x2fd
[pbf2json]:
[pbf2json]: goroutine 8 [running]:
[pbf2json]:     goroutine running on other thread; stack unavailable
[pbf2json]: created by github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:151 +0x2fd
[pbf2json]:
[pbf2json]: goroutine 9 [chan send]:
[pbf2json]: github.com/qedus/osmpbf.(*Decoder).Start.func2(0xc820018320, 0x4, 0xc820032030, 0xc820032038, 0xc8200167d0)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:170 +0x273
[pbf2json]: created by github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:180 +0x408
[pbf2json]:
[pbf2json]: goroutine 10 [chan receive]:
[pbf2json]: github.com/qedus/osmpbf.(*Decoder).Start.func3(0xc820018320, 0x4)
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:188 +0xd2
[pbf2json]: created by github.com/qedus/osmpbf.(*Decoder).Start
[pbf2json]:     /home/peter/.go/src/github.com/qedus/osmpbf/decode.go:202 +0x43a
[pbf2json]:
[pbf2json]: goroutine 17 [select]:
[pbf2json]: github.com/syndtr/goleveldb/leveldb/util.(*BufferPool).drain(0xc8201f2000)
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:206 +0x29e
[pbf2json]: created by github.com/syndtr/goleveldb/leveldb/util.NewBufferPool
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/util/buffer_pool.go:237 +0x270
[pbf2json]:
[pbf2json]: goroutine 11 [select]:
[pbf2json]: github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc8201f4180)
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:90 +0x54d
[pbf2json]: created by github.com/syndtr/goleveldb/leveldb.openDB
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db.go:139 +0x65f
[pbf2json]:
[pbf2json]: goroutine 12 [select]:
[pbf2json]: github.com/syndtr/goleveldb/leveldb.(*DB).mpoolDrain(0xc8201f4180)
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db_state.go:96 +0x22d
[pbf2json]: created by github.com/syndtr/goleveldb/leveldb.openDB
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db.go:140 +0x681
[pbf2json]:
[pbf2json]: goroutine 13 [select]:
[pbf2json]: github.com/syndtr/goleveldb/leveldb.(*DB).tCompaction(0xc8201f4180)
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:804 +0x816
[pbf2json]: created by github.com/syndtr/goleveldb/leveldb.openDB
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db.go:146 +0x881
[pbf2json]:
[pbf2json]: goroutine 14 [select]:
[pbf2json]: github.com/syndtr/goleveldb/leveldb.(*DB).mCompaction(0xc8201f4180)
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:751 +0x254
[pbf2json]: created by github.com/syndtr/goleveldb/leveldb.openDB
[pbf2json]:     /home/peter/.go/src/github.com/syndtr/goleveldb/leveldb/db.go:147 +0x8a3
[pbf2json]: [signal 0xb code=0x1 addr=0xf0f5413dc6a pc=0x1287b]
[pbf2json]: runtime stack:
[pbf2json]: runtime.throw(0x2ec720, 0x2a)
[pbf2json]:     /usr/local/go/src/runtime/panic.go:547 +0x90
[pbf2json]: runtime.sigpanic()
[pbf2json]:     /usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a
[pbf2json]: runtime.unlock(0x3f3ea0)
[pbf2json]:     /usr/local/go/src/runtime/lock_sema.go:107 +0x14b
[pbf2json]: runtime.(*mheap).alloc_m(0x3f3ea0, 0x1, 0x8, 0x1b566f8)
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:492 +0x314
[pbf2json]: runtime.(*mheap).alloc.func1()
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:502 +0x41
[pbf2json]: runtime.systemstack(0xc8201d5e58)
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:307 +0xab
[pbf2json]: runtime.(*mheap).alloc(0x3f3ea0, 0x1, 0x10000000008, 0x1251f)
[pbf2json]:     /usr/local/go/src/runtime/mheap.go:503 +0x63
[pbf2json]: runtime.(*mcentral).grow(0x3f54f0, 0x0)
[pbf2json]:     /usr/local/go/src/runtime/mcentral.go:209 +0x93
[pbf2json]: runtime.(*mcentral).cacheSpan(0x3f54f0, 0x1b566f8)
[pbf2json]:     /usr/local/go/src/runtime/mcentral.go:89 +0x47d
[pbf2json]: runtime.(*mcache).refill(0x465000, 0xc800000008, 0x1b566f8)
[pbf2json]:     /usr/local/go/src/runtime/mcache.go:119 +0xcc
[pbf2json]: runtime.mallocgc.func2()
[pbf2json]:     /usr/local/go/src/runtime/malloc.go:642 +0x2b
[pbf2json]: runtime.systemstack(0xc820023500)
[pbf2json]:     /usr/local/go/src/runtime/asm_amd64.s:291 +0x79
[pbf2json]: runtime.mstart()
[pbf2json]:     /usr/local/go/src/runtime/proc.go:1051
2017-06-06T19:35:31.269Z - info: [wof-pip-service:master] macroregion worker loaded 83 features in 9.76 seconds
2017-06-06T19:36:10.350Z - info: [wof-pip-service:master] region worker loaded 4870 features in 48.816 seconds
2017-06-06T19:36:12.863Z - info: [wof-pip-service:master] country worker loaded 217 features in 51.37 seconds
2017-06-06T19:37:07.331Z - info: [wof-pip-service:master] neighbourhood worker loaded 64664 features in 105.907 seconds
2017-06-06T19:37:32.033Z - info: [wof-pip-service:master] localadmin worker loaded 115187 features in 130.434 seconds
2017-06-06T19:37:44.016Z - info: [wof-pip-service:master] county worker loaded 40926 features in 142.415 seconds
2017-06-06T19:37:59.830Z - info: [wof-pip-service:master] locality worker loaded 207861 features in 158.245 seconds
2017-06-06T19:37:59.830Z - info: [wof-pip-service:master] PIP Service Loading Completed!!!
^C
missinglink commented 7 years ago

hey @tigerlily-he I'm running linux locally, could you please provide some more info:

as per comments in https://github.com/pelias/config/issues/58#issuecomment-303099472

$ shasum build/*
$ file build/pbf2json.darwin-x64

[edit] hold off on doing anything with libleveldb, let's try the binaries recompiled against go 1.8.3 first.

it's possible that the library has a dependency on the C lib libleveldb, if possible, can you please try installing that and trying again?

in ubuntu it's provided by these packages:

$ dpkg -l | grep leveldb
ii  libleveldb-dev:amd64                            1.18-5                                        amd64        fast key-value storage library (development files)
ii  libleveldb1v5:amd64                             1.18-5                                        amd64        fast key-value storage library
missinglink commented 7 years ago

this might also be related https://github.com/golang/go/issues/17367, which version of OSX are you running?

I compiled this with:

$ go version
go version go1.6.2 linux/amd64

I'll upgrade and recompile

missinglink commented 7 years ago

ok, I recompiled against go 1.8.3 and pushed the binaries up to the recompile branch.

the new checksums are:

$ shasum build/*
3c53771424f7ddde282c22dcd393edf9ec600b72  build/pbf2json.darwin-x64
3ba1c9c28b9330d760c10db2cc5ea386912daf85  build/pbf2json.linux-arm
7c585c7dfec02279779b2e12a9958d865b8b8c28  build/pbf2json.linux-x64
9acde4aa9faaf7056c211632164ed988e4100604  build/pbf2json.win32-x64

please try 3c53771424f7ddde282c22dcd393edf9ec600b72