omniscale / imposm3

Imposm imports OpenStreetMap data into PostGIS
http://imposm.org/docs/imposm3/latest/
Apache License 2.0
719 stars 157 forks source link

parser: skip parsing some primative groups #111

Closed missinglink closed 8 years ago

missinglink commented 8 years ago

hey, I could be wrong about this..

it seems that if you pass nil channels to the parser like this imposm.NewParser(file, nil, nil, nil, relations) that all the nodes and way blocks will be decoded anyway?

is that correct? if so I would be happy to PR in order to only parse the blocks with valid channels

missinglink commented 8 years ago

something like this? https://github.com/omniscale/imposm3/compare/master...missinglink:ignore_nil_chans

I have a script which does 3 passes over the pbf to extract boundary relations.

First time it only reads the relations and stores the way ids in a bitmask, then on the second pass it reads the ways and stores the node refs in another bitmask and on the final pass it assembles the relations.

it seems like the diff above improves performance in this scenario (nyc sized extract)

# patch
real    0m12.877s
user    0m59.048s
sys     0m2.604s

# master
real    0m20.926s
user    1m37.308s
sys     0m3.540s
olt commented 8 years ago

That patch parses the stringtable 3 times. I can merge it, if you create a PR where this is fixed.

missinglink commented 8 years ago

will PR a cleaner version