MapShed Job Stub

[rajadain]

Overview

Extracts common functionality into a trait that can be mixed in with multiple Spark Job implementations. Adds a simple stub for a MapshedJob which currently simply returns the the count of cells grouped by their NLCD value for a given polygon.

Testing Instructions

1. Package the code:

   ./sbt "project summary" assembly

2. Start the worker VM. Within the VM, rename /opt/geoprocessing/mmw-geoprocessing-0.4.0.jar to /opt/geoprocessing/mmw-geoprocessing-0.4.0.jar.bak
3. Copy the generated summary/target/scala-2.10/mmw-geoprocessing-assembly-0.4.0.jar file into the worker VM as /opt/geoprocessing/mmw-geoprocessing-0.4.0.jar
4. Copy the two test requests into your worker VM: MapshedJob.json.txt and SummaryJob.json.txt

5. Ensure that line 34 of /opt/spark-jobserver/spark-jobserver.conf reads "geotrellis.spark.io.kryo.KryoRegistrator":

   vagrant@worker:~$ sed '34q;d' /opt/spark-jobserver/spark-jobserver.conf 
    context-settings.passthrough.spark.kryo.registrator = "geotrellis.spark.io.kryo.KryoRegistrator"

   This is needed for the newer version of GeoTrellis.

6. Restart Spark Job Server

   vagrant@worker:~$ sudo restart spark-jobserver
   spark-jobserver start/running, process 3245

7. Create a job context:

   vagrant@worker:~$ curl --silent --data '' 'http://localhost:8090/contexts/geoprocessing-context'
   OK

8. Test MapshedJob:

   vagrant@worker:~$ curl --silent --data-binary @MapshedJob.json.txt 'http://localhost:8090/jobs?sync=true&context=geoprocessing-context&appName=geoprocessing-0.4.0&classPath=org.wikiwatershed.mmw.geoprocessing.MapshedJob'
   {
    "result": [{
      "11": 11861,
      "90": 25355,
      "22": 31,
      "21": 1139,
      "43": 99663,
      "71": 2326,
      "31": 751,
      "95": 1534,
      "42": 54230,
      "41": 77316,
      "52": 25967
    }]
   }

9. Test SummaryJob to ensure it still works correctly:

   vagrant@worker:~$ curl --silent --data-binary @SummaryJob.json.txt 'http://localhost:8090/jobs?sync=true&context=geoprocessing-context&appName=geoprocessing-0.4.0&classPath=org.wikiwatershed.mmw.geoprocessing.SummaryJob'
   {
    "result": [{
      "(71,3)": 865,
      "(90,3)": 1650,
      "(22,7)": 5,
      "(11,1)": 104,
      "(11,6)": 10,
      "(52,3)": 7051,
      "(22,3)": 7,
      "(22,1)": 6,
      "(11,7)": 276,
      "(90,2)": 644,
      "(21,3)": 89,
      "(11,3)": 10998,
      "(31,7)": 208,
      "(52,6)": 40,
      "(52,2)": 1213,
      "(90,1)": 1901,
      "(43,4)": 40799,
      "(11,2)": 81,
      "(21,7)": 235,
      "(42,6)": 63,
      "(95,3)": 144,
      "(71,7)": 242,
      "(90,7)": 7473,
      "(71,1)": 207,
      "(95,2)": 8,
      "(52,7)": 5897,
      "(41,1)": 7486,
      "(22,4)": 13,
      "(41,7)": 10741,
      "(21,4)": 434,
      "(31,4)": 254,
      "(42,7)": 12392,
      "(43,3)": 21667,
      "(71,6)": 17,
      "(41,2)": 1826,
      "(52,1)": 2476,
      "(90,6)": 2324,
      "(21,2)": 19,
      "(95,7)": 471,
      "(42,4)": 30492,
      "(21,1)": 297,
      "(71,2)": 198,
      "(41,6)": 245,
      "(21,6)": 65,
      "(42,1)": 4790,
      "(43,6)": 380,
      "(43,7)": 19607,
      "(41,4)": 32622,
      "(52,4)": 9290,
      "(41,3)": 24396,
      "(71,4)": 797,
      "(42,3)": 4531,
      "(95,6)": 112,
      "(95,1)": 68,
      "(95,4)": 731,
      "(43,1)": 13473,
      "(90,4)": 11363,
      "(42,2)": 1962,
      "(11,4)": 392,
      "(31,2)": 93,
      "(31,3)": 100,
      "(31,1)": 96,
      "(43,2)": 3737
    }]
   }

Connects #25

[lewfish]

+1, tested. Good testing instructions.