search

qedus / osmpbf

OpenStreetMap PBF file format parser in Go Lang.
MIT License
141 stars 29 forks source link

Parse only for specific element types #34

Open KonradBreitsprecher opened 5 years ago

KonradBreitsprecher commented 5 years ago

Thanks for this nice package! I'm using it for a routing application that is only interested in the nodes that appear in ways. To filter this and not run into memory problems, my approaching was to scan a file twice; first for the ways, then again only for the relevant nodes. Is there an easy workaround to skip the way/node/relation blocks entirely?

jongillham commented 4 years ago

Hi @KonradBreitsprecher unfortunately you need to parse the PBF file from beginning to end and just ignore the blocks you don't want. I wonder if it's possible just to do one parse of the file and store the information you need as you go in order to be referenced later in the PBF file?

KonradBreitsprecher commented 4 years ago

Ok I ended up skipping the blocks in parsePrimitiveGroup(). If I keep track of all the IDs that appear in ways and skip those nodes in the second run (with a lookup map[int64]struct{}), I'm already a little bit faster than collecting all the nodes. At least for the germany pbf. I was also wondering if I could reuse the decoder and jump back to the beginning of the file?