Add benchmarking script

kellyi commented 7 years ago

Overview

This PR adds a benchmark script along with some sample input data to the project. I originally tried writing this in Node but for a few reasons noted in the comments below it turned out not to be a great candidate for this. I ultimately used Python instead.

This includes a dependency on the requests library which is installed on the Worker VM and can be installed locally using easy_install requests. If it's not available, the script exits immediately with a note that it's required.

Connects #66

Demo

Here's the output from running the script for all included operations:

Timing RasterGroupedAverage ->

RasterGroupedAverage, run 1 -> 0.39802 s
RasterGroupedAverage, run 2 -> 0.332305 s
RasterGroupedAverage, run 3 -> 0.468983 s
RasterGroupedAverage, run 4 -> 0.37525 s
RasterGroupedAverage, run 5 -> 0.321613 s

RasterGroupedAverage average -> 0.3792342 s

Timing HUC12 RasterGroupedCount, 1 layers ->

HUC12 RasterGroupedCount, 1 layers, run 1 -> 0.431287 s
HUC12 RasterGroupedCount, 1 layers, run 2 -> 0.327095 s
HUC12 RasterGroupedCount, 1 layers, run 3 -> 1.382664 s
HUC12 RasterGroupedCount, 1 layers, run 4 -> 0.31057 s
HUC12 RasterGroupedCount, 1 layers, run 5 -> 1.193688 s

HUC12 RasterGroupedCount, 1 layers average -> 0.7290608 s

Timing HUC12 RasterGroupedCount, 2 layers ->

HUC12 RasterGroupedCount, 2 layers, run 1 -> 0.515212 s
HUC12 RasterGroupedCount, 2 layers, run 2 -> 0.586832 s
HUC12 RasterGroupedCount, 2 layers, run 3 -> 0.402033 s
HUC12 RasterGroupedCount, 2 layers, run 4 -> 0.364732 s
HUC12 RasterGroupedCount, 2 layers, run 5 -> 0.368958 s

HUC12 RasterGroupedCount, 2 layers average -> 0.4475534 s

Timing HUC12 RasterGroupedCount, 3 layers ->

HUC12 RasterGroupedCount, 3 layers, run 1 -> 0.582437 s
HUC12 RasterGroupedCount, 3 layers, run 2 -> 0.360945 s
HUC12 RasterGroupedCount, 3 layers, run 3 -> 0.422739 s
HUC12 RasterGroupedCount, 3 layers, run 4 -> 0.329596 s
HUC12 RasterGroupedCount, 3 layers, run 5 -> 0.489557 s

HUC12 RasterGroupedCount, 3 layers average -> 0.4370548 s

Timing HUC10 RasterGroupedCount, 1 layers ->

HUC10 RasterGroupedCount, 1 layers, run 1 -> 0.49734 s
HUC10 RasterGroupedCount, 1 layers, run 2 -> 0.510371 s
HUC10 RasterGroupedCount, 1 layers, run 3 -> 0.774718 s
HUC10 RasterGroupedCount, 1 layers, run 4 -> 0.826413 s
HUC10 RasterGroupedCount, 1 layers, run 5 -> 0.665792 s

HUC10 RasterGroupedCount, 1 layers average -> 0.6549268 s

Timing HUC10 RasterGroupedCount, 2 layers ->

HUC10 RasterGroupedCount, 2 layers, run 1 -> 1.117139 s
HUC10 RasterGroupedCount, 2 layers, run 2 -> 1.839658 s
HUC10 RasterGroupedCount, 2 layers, run 3 -> 1.038473 s
HUC10 RasterGroupedCount, 2 layers, run 4 -> 0.887687 s
HUC10 RasterGroupedCount, 2 layers, run 5 -> 0.887977 s

HUC10 RasterGroupedCount, 2 layers average -> 1.1541868 s

Timing HUC10 RasterGroupedCount, 3 layers ->

HUC10 RasterGroupedCount, 3 layers, run 1 -> 1.066983 s
HUC10 RasterGroupedCount, 3 layers, run 2 -> 1.129731 s
HUC10 RasterGroupedCount, 3 layers, run 3 -> 1.152303 s
HUC10 RasterGroupedCount, 3 layers, run 4 -> 1.217285 s
HUC10 RasterGroupedCount, 3 layers, run 5 -> 1.08798 s

HUC10 RasterGroupedCount, 3 layers average -> 1.1308564 s

Timing HUC8 RasterGroupedCount, 1 layers ->

HUC8 RasterGroupedCount, 1 layers, run 1 -> 1.744785 s
HUC8 RasterGroupedCount, 1 layers, run 2 -> 1.6372 s
HUC8 RasterGroupedCount, 1 layers, run 3 -> 1.43584 s
HUC8 RasterGroupedCount, 1 layers, run 4 -> 1.694609 s
HUC8 RasterGroupedCount, 1 layers, run 5 -> 1.565295 s

HUC8 RasterGroupedCount, 1 layers average -> 1.6155458 s

Timing HUC8 RasterGroupedCount, 2 layers ->

HUC8 RasterGroupedCount, 2 layers, run 1 -> 2.352573 s
HUC8 RasterGroupedCount, 2 layers, run 2 -> 2.290398 s
HUC8 RasterGroupedCount, 2 layers, run 3 -> 2.584728 s
HUC8 RasterGroupedCount, 2 layers, run 4 -> 2.465196 s
HUC8 RasterGroupedCount, 2 layers, run 5 -> 2.519123 s

HUC8 RasterGroupedCount, 2 layers average -> 2.4424036 s

Timing HUC8 RasterGroupedCount, 3 layers ->

HUC8 RasterGroupedCount, 3 layers, run 1 -> 3.264991 s
HUC8 RasterGroupedCount, 3 layers, run 2 -> 3.30156 s
HUC8 RasterGroupedCount, 3 layers, run 3 -> 3.804493 s
HUC8 RasterGroupedCount, 3 layers, run 4 -> 2.875721 s
HUC8 RasterGroupedCount, 3 layers, run 5 -> 2.801823 s

HUC8 RasterGroupedCount, 3 layers average -> 3.2097176 s

Notes

I used the built in ArgumentParser library for the command line options. Running ./scripts/benchmark with no options given will run all the timing jobs; running it with --help will print a usage string. You can also run it with --huc8 --huc12 --huc10 --rga and it will execute whichever of the jobs you're provided as arguments.

If it's not available globally, you'll need to sudo easy_install requests.

Testing

get this branch, then run ./scripts/server
in another terminal pane run ./scripts/benchmark --help and verify that you see the usage string
run ./scripts/benchmark with no options and verify that all the timing jobs run
run it with combinations of --huc8 --huc12 --huc10 --rga and verify that it executes the jobs you've specified.

mmcfarland commented 7 years ago

A general note, it's a little unfortunate that the script is written in node, since the worker machine that runs this service doesn't have that runtime installed. That'll make it slightly challenging to run this on an instance built off of a CI built AMI, or even in our dev VM setup - even if it works fine on your dev host. The benchmarking will be most meaningful on instances resembling our production infrastructure, due to the compute configuration but mostly due to the network available.

I don't think it's problematic enough it to rewrite, but it's a good practice to keep these utility scripts in bash or python, which is both consistent with our other scripts and likely to be available on most of our setups without augmentation.

kellyi commented 7 years ago

it's a little unfortunate that the script is written in node, since the worker machine that runs this service doesn't have that runtime installed

Good point. There was another issue I encountered while trying to use Node which was that the requests were asynchronous and getting them to complete in the correct order required lacing Promises throughout the script in a way that didn't seem like it'd be very easy to maintain.

To simplify things, 269be61 replaces the Node.js version with a simpler Python script. Going to write a proper PR message with testing instructions now.

kellyi commented 7 years ago

Thanks for taking a look at this!

WikiWatershed / mmw-geoprocessing