search

azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

Parse countries.geojson once #81

Closed mojodna closed 6 years ago

mojodna commented 6 years ago

When running in local mode, Countries.all being a def results in countries.geojson being parsed at the beginning of each task execution (and concurrently to the tune of the number of cores available). This allows parsing to occur once over the lifetime of the job.

This could use some general cleanup and also needs to be tested on a proper Spark cluster before being merged.

Refs #77

mojodna commented 6 years ago

This does indeed work on a proper Spark cluster. Each executor initializes Countries.all once when it runs the first task.