Add Support for .osm.pbf Input

sqrtM commented 11 months ago

Summary

This pull request introduces support for the PBF input format.

Changes Made

Added support for reading PBF data.
Added osmpbf dependency

Issue

Closes #225

Testing

Added virgin_islands folder to the test suite. It is large enough to show the capability and speed of the PBF format over XML, while not being so large that it becomes inhibitive for the developer (or Github size limits).

Screenshots

Main island visualized on geojson.io using the generated geometry.

Additional Notes

There are a couple of considerations that are important to take note of here. If any of them are not to your liking, just let me know, and I'll fix them right up! :

PBF is NOT supported in WASM as of this pull request. This should be handled in a seperate issue if we wish to implement it.
When deciding the bounding box for the pbf input, the priority is as follows:
1. Check for a bounding box in the blob header
2. If a bounding box does not exist in the header, use the data from boundaries.json
3. If boundaries.json does not exist, scrape bounds from nodes directly.
4. If scrape fails, panic.
Similarly, this functionality is making use of a BlobReader rather than the par_map_reduce function. This was done to get access to the more accurate bounding box of the header without needing the user to supply one or scrape.
And finally, just to be clear, the "additions" number is so large because of the large files in the virgin_islands directory.

Thank you very much for your time and have a nice day!

dabreegster commented 11 months ago

Thank you so much for the PR! Just to let you know, I'm rushing to finish some urgent work before State of the Map next weekend. I don't think I'll be able to look at this til the 13th. Sorry for the delay :\

sqrtM commented 11 months ago

Thank you so much for the PR! Just to let you know, I'm rushing to finish some urgent work before State of the Map next weekend. I don't think I'll be able to look at this til the 13th. Sorry for the delay :\

oh, man, no worries. get to it when you get to it and I'll be around to clean up after the review. not a problem in the least. have a great week!

sqrtM commented 10 months ago

great! thanks a bunch for the review ! :)

gonna be a BIT busy with the day job this week, but this should be done sooner than later!

be back soon with the changes + rebase. have a nice rest of the week!

sqrtM commented 10 months ago

alright -- as for your current suggestions, i think everything should be implemented! feel free to have another glance over everything at your leisure.

ran cargo fmt
shrunk the pbf test down to just a few blocks
removed helper function
fixed my silly commenting-out of the bail! (sorry about missing that -- have a look at how I worked around that in the new version. it doesn't bail, but it does print an error and return unit value. that seemed like the most sensible solution. let me know if you'd prefer something else!)
switched the match statement to an if

let me know if there's anything else you'd like or any other suggestions you may have. thanks!! (PS sorry for accidentally closing the pr. i'm honestly not entirely sure what caused that. but everything should be good!)

sqrtM commented 10 months ago

Happy to help! let me know if there's anything else i can do for you. thanks!

a-b-street / osm2streets