koopjs / koop-provider-file-geojson

Simple Koop provider for serving file-based GeoJSON
Other
5 stars 6 forks source link

How to resolve the no "idField" assignment warning? #4

Closed bmulcahy closed 6 years ago

bmulcahy commented 6 years ago
WARNING: requested provider has no "idField" assignment. This can cause errors in ArcGIS clients

What is the correct way to resolve this warning? How should our data be formatted?

rgwozdz commented 6 years ago

This warning stems from the need for ArcGIS clients usually requiring an OBJECTID field. In addition to requiring this field, OBJECTIDs that are not integers or outside the range of 0 - 2,147,483,647 can break features in some ArcGIS clients.

How you resolve this warning depends on your data. If your data already has a field that can serve as a unique ID and is an integer in the range of 0 - 2,147,483,647, you can instruct koop to use it as the OBJECTID by adding idField:<your-field-name> to the geojson metadata set in your provider.

If you don't set idField and don't have a field called OBJECTID in your data, koop will create one using a numeric hash of the feature, scaled to a range of 0 - 2,147,483,647. There is a chance of collisions using this method (perhaps a small one) - hence the warning you are seeing.

bmulcahy commented 6 years ago

Thanks for the explanation. Could we add this to the docs or readme?

rgwozdz commented 6 years ago

@bmulcahy if you go to this page and look in the "Metadata" section, you'll see a similar explanation. I'll admit that it is somewhat buried - we're in the early stages of reorganizing our docs. Hope to publish a new site soon.

bmulcahy commented 6 years ago

Ah my fault I should have looked closer

simon-fisher commented 5 years ago

I am getting this warning, but don't want to assign an idField and just want to let koop create the objectid field. I do see this being created successfully. However I can't seem to query using the objectid. ie. where=OBJECTID=1234567. No features are returned. Is this expceted? Do I need to provide my own objectid field?

rgwozdz commented 5 years ago

No, not expected. I think you have found a bug, but it's with koop-core. Working to fix it now.

rgwozdz commented 5 years ago

@simon-fisher - see https://github.com/koopjs/winnow/issues/115

rgwozdz commented 5 years ago

@simon-fisher - I think I have fixed your noted issue using OBJECTID in the where parameter when those OBJECTIDs are created by Koop. You will need to upgrade your deployment so that Koop uses newly release Winnow v1.16.7.

simon-fisher commented 4 years ago

Sorry I just came back to this now. Even after upgrading winnow I still get the warning and OBJECTID where queries fail. Is there any chance this issue came back? Or do I need to upgrade koop to latest version as well?

rgwozdz commented 4 years ago

I'd upgrade everything first, then check. Please let me know if you still have issues

simon-fisher commented 4 years ago

I upgraded everything to latest, koop at 3.17.1. and same issue appears.

Not sure if this is related, but when searching through for winnow I did notice FeatureServer still uses "winnow": "^1.16.5" https://github.com/koopjs/FeatureServer/blob/master/package.json but elsewhere it references 1.16.11

rgwozdz commented 4 years ago

It's fine that the package uses ^1.16.5; the ^ should ensure the most recent minor release is installed. You can double check by looking in the lock file or the node_modules/winnow file.

I think you may have some aspect of implementation wrong. I just tested this with the socrata provider, and the request http://localhost:8080/koop-provider-socrata/data.ct.gov/y6p2-px98/FeatureServer/0/query?where=OBJECTID=2096926232 returned the expected feature. My advice would be to do the same, i.e., create a fresh Koop instance using the socrata provider and attempt the same request - just so we know we're on the same page. Then I'd ask you to shared your provider code and sample of your geojson; maybe I will spot something.

simon-fisher commented 4 years ago

I have been playing around with this some and it seems I can get some simple queries using OBJECTID to work, as well as the orderByFields works with OBJECTID. Some more complex queries don't work with OBJECTID, including a Sql IN statement.

Example: https://localhostdev/my-provider/client1/FeatureServer/0/query/?id=parcels&where=OBJECTID in (4767223,2083799)&orderByFields=OBJECTID

Does Sql IN work for you?

rgwozdz commented 4 years ago

IN is currently not supported for OBJECTID queries. See winnow. Is there any way you can define your own field for the unique identifier (using idField to point to it)? It's probably the most reliable solution. Otherwise you could try to PR support for IN.

simon-fisher commented 4 years ago

That's actually what I am trying to do now. I added a field named OBJECTID to my data, but I get 2 fields, mine as objectid and the koop generated field as OBJECTID. I guess I need to code or configure koop to use mine correct?

rgwozdz commented 4 years ago

Can you paste a sample of your geojson?

simon-fisher commented 4 years ago

{ "objectIdFieldName": "OBJECTID", "uniqueIdField": { "name": "OBJECTID", "isSystemMaintained": true }, "globalIdFieldName": "", "hasZ": false, "hasM": false, "spatialReference": { "wkid": 4326 }, "fields": [ { "name": "OBJECTID", "type": "esriFieldTypeOID", "alias": "OBJECTID", "sqlType": "sqlTypeInteger", "domain": null, "defaultValue": null }, { "name": "objectid", "type": "esriFieldTypeInteger", "alias": "objectid", "sqlType": "sqlTypeOther", "domain": null, "defaultValue": null }, { "name": "parid", "type": "esriFieldTypeString", "alias": "parid", "sqlType": "sqlTypeOther", "domain": null, "defaultValue": null, "length": 128 }, { "name": "jur", "type": "esriFieldTypeString", "alias": "jur", "sqlType": "sqlTypeOther", "domain": null, "defaultValue": null, "length": 128 }, { "name": "taxyr", "type": "esriFieldTypeInteger", "alias": "taxyr", "sqlType": "sqlTypeOther", "domain": null, "defaultValue": null }, { "name": "gispin", "type": "esriFieldTypeString", "alias": "gispin", "sqlType": "sqlTypeOther", "domain": null, "defaultValue": null, "length": 128 } ], "features": [ { "attributes": { "objectid": 4, "parid": "001001 E00006", "jur": "000", "taxyr": 2008, "gispin": "001001 E00006", "OBJECTID": 343002794 }, "geometry": { "x": 0, "y": 0 } }, { "attributes": { "objectid": 5, "parid": "001001 E00007", "jur": "000", "taxyr": 2008, "gispin": "001001 E00007", "OBJECTID": 739650346 }, "geometry": { "x": 0, "y": 0 } } ], "exceededTransferLimit": false, "geometryType": "esriGeometryPoint" }

rgwozdz commented 4 years ago

Sorry, I mean the geojson that your provider is passing into koop (into the the callback of the getData function); not the Esri JSON that ends up being served by Koop

simon-fisher commented 4 years ago

I figured it out by setting metadata.idField to my OBJECTID. All OBJECTID queries seem to work now including Sql IN. I think I'm all good now.

Thanks so much for your help on this. Simon