Issue working with polygons

[hhkaos]

Hi @dmfenton,

I'm playing a little bit with this provider but I'm having some issues:

1. I'm trying to match the name of the "Provincia" (~state) of each row of this spreadsheet with a boundaries file I have on my local. And the geojson generated seems to be OK.
2. Now when I query the provider the output is this, which at first seems to be OK too.
3. But when I try to load it in the webmap viewer it is not displaying the geometries as you can see here:

Because I wasn't sure if the problem was with the way I was generating the geometries, I also tested the output with geojsonhint and it only displayed a warning Polygons and MultiPolygons should follow the right-hand rule, so I continued testing:

4. I uploaded the GeoJSON I was passing to callback(null, geojson) though the "Content" tab to generate a new item with the hosted feature service
5. And, as I expected, from the visualization tab it couldn't see the geometries either
6. But the weird thing comes now, if I load it to the webmap viewer directly, it worked 😮

So, I have no idea on what's happening, is there any way I can test/debug the output from Koop? Something like a "FeatureService linter", or do you have any clue on what's going on?

By the way, I also tried to use "the Exporter" but it doesn't work here, right?

These are the modifications I have done:

const google = require('googleapis')
const config = require('./config/production.json')
const provincias = require('./output.json')
const sheets = google.sheets('v4')
const XVARS = ['x', 'longitude', 'lon', 'longitud', 'Lon']
const YVARS = ['y', 'latitude', 'lat', 'latitud', 'Lat']

function GoogleSheets () {}

GoogleSheets.prototype.getData = function getData (req, callback) {
  const spreadsheetId = req.params.host  // e.g. 1JlPaiuIHXmkfpLBaQdoRixPSasjX5NlDte70pyFT9yI OR 1dK_touGylnTtJBzve2HEwfev_f6JxCpRMb2NZ-LMI1g  ::Providers have built-in support for capturing request params, aka. googlesheets/:host/:id/FeatureServer/0
  const range = req.params.id // e.g. Park Cleanup!A1:H  OR World Cities!A1:I
  const gsOpts = {
    auth: config.googlesheets.auth,
    spreadsheetId, // e.g. https://docs.google.com/spreadsheets/d/1JlPaiuIHXmkfpLBaQdoRixPSasjX5NlDte70pyFT9yI/edit?usp=sharing
    range
  }
  sheets.spreadsheets.values.get(gsOpts, (err, res) => {
    if (err) return callback(err)
    const geojson = translate(res)
    geojson.ttl = config.googlesheets.ttl || 1200 // 20 minutes
    geojson.metadata = {
      name: range.split('!')[0], // Get the workbook name before ! symbol and set as layer name
      description: 'Collaborate in Google docs, analyse in ArcGIS'
    }

    var fs = require('fs');
    fs.writeFile("result.json", JSON.stringify(geojson, null, 2), function(err) {
        if(err) {
            return console.log(err);
        }

        console.log("The file was saved!");
    });

    callback(null, geojson)
  })
}

function translate (response) {
  const propertyNames = createPropertyNames(response.values[0])
  return {
    type: 'FeatureCollection',
    features: response.values.slice(1).map(row => { return formatFeature(row, propertyNames) })
  }
}

function formatFeature (row, propertyNames) {
  const x = propertyNames.indexOf('x')
  const y = propertyNames.indexOf('y')
  if(x != -1 && y != -1){
      return {
        type: 'Feature',
        geometry: {
          type: 'Point',
          coordinates: [parseFloat(row[x]), parseFloat(row[y])] // Make sure coordinates are numbers not strings
        },
        properties: row.reduce((props, prop, i) => {
          if (i !== x && i !== y) props[propertyNames[i]] = prop
          return props
        }, {})
      }
  }else{

      const coordinates = propertyNames.indexOf('PROVINCIAS')
      var slugyField = slugify(row[coordinates]);

      switch(slugyField){
          case 'baleares':
          slugyField = 'balears-illes';
          break;
          case 'las-palmas':
          slugyField = 'palmas-las';
          break;
          case 'sta-cruz':
          slugyField = 'santa-cruz-de-tenerife';
          break;
          case 'araba':
          slugyField = 'arabalava';
          break;
          case 'a-corua':
          slugyField = 'corua-a';
          break;
          case 'orense':
          slugyField = 'ourense';
          break;
          case 'la-rioja':
          slugyField = 'rioja-la';
          break;
          case 'alicante':
          slugyField = 'alicantealacant';
          break;
          case 'castelln':
          slugyField = 'castellncastell';
          break;
          case 'valencia':
          slugyField = 'valenciavalncia';
          break;

      }
      if(!provincias[slugyField]){
          console.log("slugyField=",slugyField)
          console.log("provincias[slugyField]]=",provincias[slugyField])
      }

      var obj = {
        type: 'Feature',
        geometry: provincias[slugyField],
        properties: row.reduce((props, prop, i) => {
          if (i !== x && i !== y) props[propertyNames[i]] = prop
          return props
        }, {})
      };

      return obj;
  }
}

function createPropertyNames (header) {
  return header.map(head => {
    const candidate = head.toLowerCase()
    if (YVARS.indexOf(candidate) > -1) return 'y'
    else if (XVARS.indexOf(candidate) > -1) return 'x'
    else return head
  })
}

function slugify(text)
{
  return text.toString().toLowerCase()
    .replace(/\s+/g, '-')           // Replace spaces with -
    .replace(/[^\w\-]+/g, '')       // Remove all non-word chars
    .replace(/\-\-+/g, '-')         // Replace multiple - with single -
    .replace(/^-+/, '')             // Trim - from start of text
    .replace(/-+$/, '');            // Trim - from end of text
}

module.exports = GoogleSheets

Thanks in advance!

[hhkaos]

Or maybe... @jgravois @slibby @ajturner can help me here ^_^

[hhkaos]

Ah, and I forgot to mention that at first I tried to load the boundaries directly into the cell and it was working with a couple of them, but then, I had a problem with the maximum number of characters I could add to a single cell (50000), that's why I started using a local file with the boundaries.

[hhkaos]

thanks Daniel :)

[rgwozdz]

@hhkaos the first thing I would do to debug is open up Chrome Dev Tools or Fiddler and inspect the requests/responses issued by the web-map-viewer to koop that return responses that fail to render. That or those requests will likely have querystring parameters that are modifying the request and have an effect on the response. Feel free to paste that request URL here and I can also take a look at the response; it's possible that something is missing from the response that the map-viewer needs; if we find what's missing we can try to track down the source of the omission.

[rgwozdz]

@hhkaos - also, upgrade your instance with yarn so that you are using FeatureServer 2.12.0 and winnow 1.15.2. That will put us at the same starting point.

[hhkaos]

Thanks Rich!, I have been out of the office but next week I'll follow your advices. Thank you very much for your help!!

[hhkaos]

Hi @rgwozdz sorry for taking so long to answer. I've updated my instance with Yarn and I have notice a warning message is showing up in the console: WARNING: requested provider has no "idField" assignment. This can cause errors in ArcGIS clients.

So I have added the idField to the metadata but I think it is still taking the OBJECTID from somewhere else.

If have also uploaded the source code to this repo and tried to deploy again the NOW service. Not sure if it worked properly, is there any way to check by URL which versions of the FeatureServer and Winnow server are been used by the service?

I have also checked the console and I can not see any error message related to this service:

The one you see is because of a Diigo extension I have installed in Chrome (but it is always there ;P).

The viewer is requesting this URLs:

• Request with: ?f=json&callback=dojo_request_script_callbacks.dojo_request_script1
• Request with: ?f=json
• Request with: ?f=json&returnIdsOnly=true&returnCountOnly=true&where=1%3D1&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=*&callback=dojo_request_script_callbacks.dojo_request_script2
• Request with: ?f=json&where=1%3D1&returnGeometry=false&spatialRel=esriSpatialRelIntersects&outFields=*&orderByFields=OBJECTID%20ASC&resultOffset=0&resultRecordCount=50&callback=dojo_request_script_callbacks.dojo_request_script3

But none seem to fail... and the service is still been identified as a table when it should be identified as a feature service 😞.

Thanks again!, Raul

[rgwozdz]

@hhkaos is the repo https://github.com/equipoimcre/09mapaeries/ private? I get a 404.

[hhkaos]

Sorry Rich, you're right, my colleagues asked me to make it private temporarily (because some dependencies need to be fixed).

[rgwozdz]

@hhkaos are you able to share your config/production.json file with me? I can help debug if you send that my way.

[hhkaos]

I have share it by email with you (because of the private key), thank you very much ! :smile:

[rgwozdz]

Looks like you need to define the geometryType in the metadata object of your provider (https://github.com/equipoimcre/09mapaeries/blob/master/sheets.js#L22-L26):

geometryType: 'MultiPolygon'

When I add that in, I get the data returned as type Feature Layer.

[hhkaos]

Awesome Rich! It looks like it is partially working but for some reason not all the geometries are been displayed. I'll check again asap. Thanks!!!

[hhkaos]

mmmm I have notice the geometries are there but they are not displayed by default, for some strange reason when I span&zoom the map different geometries become visible as you can see here:

Do you know why? :worried:

[rgwozdz]

That's odd. I'll take a look.

[rgwozdz]

@hhkaos - The data in your idField column are not being cast as integers in your provider; rather, they arrive from google-sheets as a character string. As a result your OBJECTIDs are also character-strings, but they need to be integers in the range of 0 to 2147483647; you should actually see a warning in the console stating this problem with your OBJECTIDs. I've seen issues like this cause rendering problems before. Try explicitly casting your idField data to an integer in the formatFeature function.

[hhkaos]

Awesome @rgwozdz it works now!! Thank you very very much!

Do you think we can improve the documentation about Koop providers to add this issues?:

• Add the idField and the geometryType to the metadata property
• And ensure the idField is an integer?

Thank you very very much Rich, I owe you a beer/coke in the next DevSummit ;D

[hhkaos]

Is the any way to clear the cache? (I'm updating the spreadsheet but I can't see the changes in the service yet) ^_^

[rgwozdz]

@hhkaos - very happy I was able to assist. Improvement of the docs is something I'm hoping to undertake very soon. The idField and geometryType metadata are noted in the Metadata section of this page but perhaps it's not quite visible enough. Any suggestion on better placement or organization of this page so that things are more clear?

As far as clearing the cache, I think you may just have to stop and restart the koop server.

[hhkaos]

You are completely right Rich.

Maybe I would add to Koop:

• Some additional checking to verify the idField in an Integer
• Some checking to validate the OBJECTID value is in the right range
• Also include links to the documentations when something fails (and/or even a "bip" sound on the console to get the developer attention)
• Even a debuging guide or tips&tricks section would be nice

I would assume many people won't carefully read many things.

I'm glad to start contributing if you point me to the right file where this enhancements should be done (I'm still not very familiar with the whole project :sweat_smile:).

Thanks!

[rgwozdz]

We have you're first two suggestions covered here: https://github.com/koopjs/winnow/blob/master/src/executeQuery.js#L84-L86. That was the warning that gave me the clue about your issue. It is just a warning though, and lets Koop continue to run.

Thanks for the feedback.

[rgwozdz]

@hhkaos - I've add this issue, suggesting we colorize the warning messages. We're always looking for contributors, so don't hesitate to take a crack at this if you'd like.