Raster roadmap

[knutole]

Hi @strk.

I will be travelling today, so won't be able to be online. Was hoping you could start having a look at raster imports on Thursday, and generally start getting familiar with the tileserver. It's due for a huge refactor, will start work on it Friday.

In the meantime, we need routes for importing raster and creating tiles from them. Currently we're tiling overlay rasters just by cutting up the image, but with additional raster support now, we want to remove this and put all rasters in PostGIS and create tiles from there. If you would start the work on this API endpoint, that would be really great, and I will join you on Friday.

See https://github.com/systemapic/wu/issues/424 for latest API.

[strk]

For the morning I made some robustness improvements to the import_shapefile.sh and import_raster.sh scripts. I understood that right now "importing a raster" does not involve calling the import_raster.sh at all, and that pile is not getting those rasters from the database either.

Will get familiar with the pile map configuration in the afternoon. If there's a document explaining the architecture of it I'll be happy to give it a read.

[strk]

I've pushed a postgis_raster branch in the wu repo, to import the rasters in the database, now switching to pile to see what it takes to have them rendered by mapnik. Will put work in a postgis_raster branch so that we can finally have both branches referenced by a postgis_raster branch in docker-systemapic -- if there's any pre-existing work in that direction please let me know.

[strk]

So I've pushed a postgis_raster branch of docker-systemapic which tracks postgis_raster branches of pile and wu submodules. The steps I moved involved using "createPostgisLayer" for both raster and vector data. Things don't fully work yet, but I got to learn some (few) of the internals.

Uploading to postgis works, creating a layer still returns an error which I wasn't able to track yet.

[strk]

Should I file a separate ticekt for the current problem I'm having with the postgis_raster branch ?

[strk]

This is the wu log at the end of the upload:

wu_1          | [ 'uploadStatus',
wu_1          |   { file_id: 'file_ecuubyypnopapesecamu',
wu_1          |     user_id: 'user-07bd2672-9df7-45b9-893f-b9b2b8877422',
wu_1          |     filename: 'SCF_MOD_2014_008.tif',
wu_1          |     timestamp: 1455816964363,
wu_1          |     status: 'Processing',
wu_1          |     size: 1573508,
wu_1          |     upload_success: true,
wu_1          |     error_code: null,
wu_1          |     error_text: null,
wu_1          |     processing_success: null,
wu_1          |     rows_count: null,
wu_1          |     import_took_ms: null,
wu_1          |     data_type: 'vector',
wu_1          |     original_format: null,
wu_1          |     table_name: null,
wu_1          |     database_name: null,
wu_1          |     uniqueIdentifier: '1573508-1444921344000-user-07bd2672-9df7-45b9-893f-b9b2b8877422-SCF_MOD_2014_008.tif',
wu_1          |     default_layer: null,
wu_1          |     default_layer_model: null } ]
wu_1          | [ 'esnure', null, [object Object] ]
wu_1          | [ '====================' ]
wu_1          | [ 'Chunked upload done' ]
wu_1          | [ 'err? ', undefined ]
wu_1          | [ 'result: ', undefined ]
wu_1          | [ '====================' ]
wu_1          | [ null ]

The reaction of the browser (from firebug) is that the/upload post gets this response:

{
  "file": {
    "_id": "56c60104b6b27b230066b6bb",
    "lastUpdated": "2016-02-18T17:36:04.659Z",
    "created": "2016-02-18T17:36:04.653Z",
    "type": "raster",
    "dataSize": "1573508",
    "originalName": "SCF_MOD_2014_008.tif",
    "name": "SCF_MOD_2014_008",
    "createdBy": "user-07bd2672-9df7-45b9-893f-b9b2b8877422",
    "uuid": "file_ecuubyypnopapesecamu",
    "__v": 0,
    "access": {
      "clients": [],
      "projects": [],
      "users": []
    },
    "data": {
      "image": {
        "crunched": []
      },
      "raster": {
        "file_id": "file_ecuubyypnopapesecamu"
      }
    },
    "styleTemplates": [],
    "format": [],
    "keywords": [],
    "files": []
  },
  "layer": null
}

And right after a POST to /error is sent with a string (bogus) payload:

"{\"message\":\"ReferenceError: defaultCartocss is not defined\",\"file\":\"https://localhost/js/src
/models/model.files.js\",\"line\":703,\"user\":\"Sandro Santilli\",\"stack\":\"Wu.Model.File<._requestDefaultRasterLayer
@https://localhost/js/src/models/model.files.js:703:4\\nWu.Model.File<._createDefaultRasterLayer@https
://localhost/js/src/models/model.files.js:681:3\\nWu.Model.File<._createDefaultLayer@https://localhost
/js/src/models/model.files.js:570:22\\nWu.Model.File<._createLayer@https://localhost/js/src/models/model
.files.js:553:3\\nWu.Chrome.Data<._onFileImported@https://localhost/js/src/chrome/data/chrome.data.js
:236:1\\nWu.Events<.fire@https://localhost/js/src/core/class.js:1309:5\\nWu.Data<._gotFile@https://localhost
/js/src/core/data.js:102:1\\nWu.Data<._getFile/xhr.onreadystatechange@https://localhost/js/src/core/data
.js:86:47\\n\",\"project\":\"test\"}"

[knutole]

{
        "message":"ReferenceError: defaultCartocss is not defined",
        "file":"https://localhost/js/src/models/model.files.js",
        "line":703,
        "user":"Sandro Santilli",
        "stack":"Wu.Model.File<._requestDefaultRasterLayer 
                @https://localhost/js/src/models/model.files.js:703:4nWu.Model.File<._createDefaultRasterLayer@https 
                ://localhost/js/src/models/model.files.js:681:3nWu.Model.File<._createDefaultLayer@https://localhost 
                /js/src/models/model.files.js:570:22nWu.Model.File<._createLayer@https://localhost/js/src/models/model 
                .files.js:553:3nWu.Chrome.Data<._onFileImported@https://localhost/js/src/chrome/data/chrome.data.js 
                :236:1nWu.Events<.fire@https://localhost/js/src/core/class.js:1309:5nWu.Data<._gotFile@https://localhost 
                /js/src/core/data.js:102:1nWu.Data<._getFile/xhr.onreadystatechange@https://localhost/js/src/core/data 
                .js:86:47n",
        "project":"test"
}

I think this line with defaultCartocss variable should be commented out. Could you check that you have https://github.com/systemapic/systemapic.js/commit/2dbf3d93d2b02c30327401f1c5d7c272cb290583?

[strk]

I think this was fixed with 213fea8cc4f44f2d1a3ab44a851edc8e710dcd00 -- will file other tickets for new issues