search

vpac-innovations / rsa

Storage and processing for geospatial raster data
GNU General Public License v3.0
0 stars 1 forks source link

Categorise web service for RSA datasets should make use of AKKA #55

Open lachlanhurst opened 7 years ago

lachlanhurst commented 7 years ago

There's a RSA web service that when called, extracts statistics and saves the results to the DB (https://github.com/vpac-innovations/rsa/blob/akka-workers/src/spatialcubeservice/src/main/java/org/vpac/web/controller/DatasetController.java#L164). This is a blocking web request, which was ok for the Python import script, but is not suitable for the import UI. It also doesn't make use of the AKKA stuff at all, so is pretty slow (single threaded).

I'd like to have this web service work the same way as the one we use to extract stats from the query (non blocking, uses AKKA, can track progress). For a quick hack I modified that function in DatasetController to pass the QueryDefinition it creates to the query function in https://github.com/vpac-innovations/rsa/blob/akka-workers/src/spatialcubeservice/src/main/java/org/vpac/web/controller/DataController.java#L769 . The hack can be found here https://github.com/vpac-innovations/rsa/commit/6be331b007e9c61138c52155e2b31fdf66e90e83.

Thing is I believe this hack worked, kinda. The problem is that the stats from running via AKKA are stored as query stats and not dataset stats (DataSetCats vs TaskCats) and this breaks Landblade. Based on my reading of the code everything AKKA does is based on it being query stats (not Dataset stats).

lachlanhurst commented 7 years ago

@z0u proposed steps

  1. Add parameters to DataController.query: datasetId, bandId
  2. Pass those parameters on to the master in the Job.Work object
  3. If datasetId and bandId are present, the DatabaseActor should write to DatasetCats instead of TaskCats
lachlanhurst commented 7 years ago

I'd probably suggest using landblade to do the upload and make the upload + import and categorise requests (via http://sydney.landblade.com/#/import). Once you've got the data imported into RSA the categorise request is just a GET request.

Something like http://rsaurl/{datasetId}/Band/{BandId}/TimeSlice/{TimeSliceId}/categorise would do it.

The latest code (29th May 2016) in the feature-improved-user-import supports uploading zip files via landblade, but the version in master requires you to unzip the tifs before upload.

Sydney test data

Really any raster file should work provided it covers the region that RSA has been setup to cover. sydney.zip

Note: if importing using the Sydney config settings make sure you have UPLOAD_RES=20 added to your env.sh file (defaults to 100m).

Victorian test data

Victoria.zip

forjin-vpac-innovations commented 7 years ago

So can I use live upload for this test?

And if I need to set UPLOAD_RES=20 on env.sh, do I need to restart lb server?