qedus / osmpbf

OpenStreetMap PBF file format parser in Go Lang.
MIT License
141 stars 29 forks source link

Concurrent decoder #3

Closed AlekSi closed 10 years ago

AlekSi commented 10 years ago

I implemented concurrent decoder of OSMData blocks. Also I added some small optimizations.

Benchmark results for russia-european-part-latest.osm.pbf. master:

$ go test -v -bench=. -cpu 1,2,4
BenchmarkDecoder           1    91509929417 ns/op
    decode_test.go:154: Done in 91.510 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679
BenchmarkDecoder-2         1    86301976850 ns/op
    decode_test.go:154: Done in 86.302 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679
BenchmarkDecoder-4         1    82508371534 ns/op
    decode_test.go:154: Done in 82.508 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679

this branch:

$ go test -v -bench=. -cpu 1,2,4
BenchmarkDecoder           1    85276857960 ns/op
    decode_test.go:199: Done in 85.277 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679
BenchmarkDecoder-2         1    56638467854 ns/op
    decode_test.go:199: Done in 56.638 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679
BenchmarkDecoder-4         1    42139943615 ns/op
    decode_test.go:199: Done in 42.140 seconds. Total: 77561149, Nodes: 70272938, Ways: 7082532, Relations: 205679

Objects are returned from Decode() in the same order as before (as they present in file).

Incompatible changes:

qedus commented 10 years ago

I have a couple of comments but think the speed improvement and architecture are great.