azavea / osmesa

OSMesa is an OpenStreetMap processing stack based on GeoTrellis and Apache Spark
Apache License 2.0
80 stars 26 forks source link

Use JTS on Spark instead of GT geometries #96

Closed moradology closed 5 years ago

moradology commented 5 years ago

JTS support on spark, provided by GeoMesa, ought to provide performance benefits over the use of GT geometries which, wrapped, will require extra memory allocation and which, stored in binary, will need to be manually serialized/deserialized whenever needed by a computation.

Sanity check on geometries read from an isle-of-man extract via ProcessOSM (counts should be the same): Before

[info] - parses isle of man nodes (1 second, 404 milliseconds)
[info]   + Nodes: 6633
[info] - parses isle of man wasys (21 seconds, 498 milliseconds)
[info]   + Ways: 27334
[info] - parses isle of man relations (18 seconds, 159 milliseconds)
[info]   + Relations: 336

After

[info] ProcessOSMTest:
[info] - parses isle of man nodes (1 second, 129 milliseconds)
[info]   + Nodes: 6633
[info] - parses isle of man wasys (20 seconds, 58 milliseconds)
[info]   + Ways: 27334
[info] - parses isle of man relations (15 seconds, 593 milliseconds)
[info]   + Relations: 343
moradology commented 5 years ago

@mojodna That package was outstanding, but should be good to go now