search

sharedstreets / sharedstreets-builder

Making maps connectable: stable, non-proprietary IDs and data standards for streets
MIT License
28 stars 12 forks source link

Give option for setting tmp dir #15

Open tomvantilburg opened 6 years ago

tomvantilburg commented 6 years ago

Currently fink uses the systems tmp dir for storing intermediate files. It would be nice when the user can give a custom tmp dir for this so it would be possible to dedicate larger disks for the process.

I know there is an option in flink taskmanager.tmp.dirs but it is not clear how to set this.

kpwebb commented 6 years ago

@tomvantilburg yes -- we've been configuring this in our planet.pbf builds as it takes about 4TB of scratch disk space to process a global tile set. Flink doesn't make this obvious/easy to change but I'll post the steps we're using in the builder docs.

That said, I'd love to know more about you're using the builder -- we've made a global z12 tile set that's hosted on S3 (we're in the process of moving it to the AWS public data sets program). I'm curious if that address your need or if you're doing something different with the app?

tomvantilburg commented 6 years ago

Does the tileset involve simplification/aggregation? Or is it just about the bounds of every tile? Would definitely be interested in a service, but for the rest just playing around.

kpwebb commented 6 years ago

Not sure I exactly follow. We're generating a complete tile set of OSM Planet as processed by the builder application (contains all the data described in https://github.com/sharedstreets/sharedstreets-ref-system) -- these references are used by the aggregation and conflation tools to link data.

tomvantilburg commented 6 years ago

Sorry for the confusion, I was thinking too complicated with vector tiles in mind where you want different combinations of data on different zoom levels (like roadclass 6 and up are only included from z13 and up, or something like that). Having this complete tile set available to play around with would be great!