Collections API: Add Benchmarking Script

WikiWatershed / mmw-geoprocessing

A Spark Job Server job for Model My Watershed geoprocessing.

Apache License 2.0

6 stars 6 forks source link

Collections API: Add Benchmarking Script #66

Closed rajadain closed 6 years ago

rajadain commented 6 years ago

Add a script to exercise the API and record times. It should run 5 identical tests for each of the following nine configurations:

RasterGroupedCount with 1 layer, 2 layers, 3 layers on a HUC-12
RasterGroupedCount with 1 layer, 2 layers, 3 layers on a HUC-10
RasterGroupedCount with 1 layer, 2 layers, 3 layers on a HUC-8

kellyi commented 6 years ago

To afford testing the performance of rasters of different sizes, we should also include a test which runs RasterGroupedAverage -- an operation which relies on an RDD storing double rather than int values.

Some preliminary work has been done to see whether larger tiles storing int8 values are more performant, and it'd be nice to be able to test whether we could create larger tiles for kfactor and slope RDDs etc.