Invalid bounds from data and all fields beside elastic generated (_type) are null

bytemedwb commented 6 years ago

Just set things up. I have elastic 5.4 as my back end. I have successfully added several million records to Elastic. I can view those record just fine in Kibana (including generating a map display). However, with the geoserver plugin (I am running 2.12.2 geoserver, and using the prebuilt binary plugin) I can not seem to access the data. When I do a compute from bounds the values come back a 0, 0, -1, -1. In the field configuration all the fields come back correctly and show the correct type. I only have a single geometry field and it selected as default.

I can save just fine and publish the layer. However, when I go to layer preview everything except the elastic generated fields (_type, etc.) and fid come back null. See the json below. Same thing happens if I select long or short names, and whether I select 1 or all of the fields.

Any thoughts on how I can debug this issue. I turned on DEVELOPER and VERBOSE logging in geoserver but nothing interesting shows up in the long.

`

{"type":"FeatureCollection","totalFeatures":10000,"features":[{"type":"Feature","id":"recordset.1b8562dd-ae72-4d8f-928b-b8672b5a12b2","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"1b8562dd-ae72-4d8f-928b-b8672b5a12b2","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}},{"type":"Feature","id":"recordset.2151ee58-cd8e-4965-915e-75f13207b4e7","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"2151ee58-cd8e-4965-915e-75f13207b4e7","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}},{"type":"Feature","id":"recordset.8196d69d-a942-4fbc-be24-47799786feef","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"8196d69d-a942-4fbc-be24-47799786feef","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}},{"type":"Feature","id":"recordset.535d770d-da9a-465a-82a2-cb242b63cf31","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"535d770d-da9a-465a-82a2-cb242b63cf31","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}},{"type":"Feature","id":"recordset.17c399c4-53f5-418d-82ac-69609c185086","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"17c399c4-53f5-418d-82ac-69609c185086","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}},{"type":"Feature","id":"recordset.7c4e5c1e-8a8d-4f40-87a9-bee01e4ed43d","geometry":null,"geometry_name":"actiongeolocation","properties":{"_id":"7c4e5c1e-8a8d-4f40-87a9-bee01e4ed43d","_index":"coalesce-oeevent","_type":"recordset","_score":null,"name":null,"objectkey":null,"source":null,"actiongeofullname":null,"actiongeotype":null,"avgtone":null}}

`

sjudeng commented 6 years ago

Try adding the below lines to your logging file (see here)

log4j.category.mil.nga.giat.data.elasticsearch=DEBUG
log4j.category.mil.nga.giat.process.elasticsearch=DEBUG

This should show the queries that are being sent to Elasticsearch and you can then execute them manually to see if there's any obvious issue with the response. What version of the plugin are you using? Are all the fields nested?

bytemedwb commented 6 years ago

Sjudeng:

Thanks. I have enabled the debug logging. The querys to retreive the data bounds or a just a GML preview are simple match_all queries. Here is the query log:

2018-05-17 12:49:26,384 DEBUG [mil.nga.giat.data.elasticsearch] - getBoundsInternal 2018-05-17 12:49:26,384 DEBUG [mil.nga.giat.data.elasticsearch] - getReaderInternal 2018-05-17 12:49:26,401 DEBUG [mil.nga.giat.data.elasticsearch] - Preparing recordset (oerepodev.incadencecorp.com:recordset) query 2018-05-17 12:49:26,401 DEBUG [mil.nga.giat.data.elasticsearch] - Unlimited maxFeatures not supported. Using default: 10000 2018-05-17 12:49:26,795 DEBUG [mil.nga.giat.data.elasticsearch] - Elasticsearch request: { "size" : 10000, "query" : {     "match_all" : { } }, "from" : 0 } 2018-05-17 12:49:31,801 DEBUG [mil.nga.giat.data.elasticsearch] - Search response: ElasticResponse[total=16190676, hits=10000, aggregations=null, scrollId=null, maxScore=1.0] Running that query via CURL produces the following results:

[root@node08 ~]# curl -k --cert "/etc/pki/java/temp.pem" -X GET "https://node08:9202/coalesce-oeevent/_search?pretty" -H 'Content-Type: application/json' -d' {     "query": {         "match_all": {}     }, "from" : 0 } ' { "took" : 159, "timed_out" : false, "_shards" : {     "total" : 5,     "successful" : 5,     "failed" : 0 }, "hits" : {     "total" : 16190676,     "max_score" : 1.0,     "hits" : [       {         "_index" : "coalesce-oeevent",         "_type" : "recordset",         "_id" : "2c2c4af1-5756-48c1-ae4b-1feac57696e6",         "_score" : 1.0,         "_source" : {           "eventrecordset.pmesiiptinfrastructure" : 0.25,           "coalesceentity.name" : "OEEvent",           "coalesceentity.datecreated" : "2018-03-11T15:11:33.985Z",           "eventrecordset.actiongeofullname" : "Massachusetts, United States",           "eventrecordset.pmesiipteconomic" : 0.25,           "coalesceentity.source" : "G2Core",           "eventrecordset.pmesiiptphysicalenvironment" : 0.5,           "eventrecordset.ontologyreference" : "Event",           "coalesceentity.version" : "0.2",           "eventrecordset.actiongeoadm2code" : "",           "eventrecordset.day" : 20180311,           "eventrecordset.quadclass" : 1,           "eventrecordset.tags" : [ ],           "eventrecordset.fractiondate" : 2018.1945,           "eventrecordset.goldsteinscale" : 1.9,           "eventrecordset.numsources" : 1,           "eventrecordset.sourceurl" : "http://nhpr.org/post/winter-storm-watch-tuesday-nhs-town-meeting-day",           "eventrecordset.globaleventid" : 737769639,           "eventrecordset.actiongeofeatureid" : "MA",           "eventrecordset.pmesiiptmilitary" : 0.25,           "coalesceentity.lastmodified" : "2018-03-11T15:11:33.996Z",           "eventrecordset.eventcode" : "042",           "eventrecordset.pmesiiptsocial" : 0.5,           "eventrecordset.pmesiiptinformation" : 0.5,           "eventrecordset.actiongeoadm1code" : "USMA",           "eventrecordset.avgtone" : -4.989154,           "eventrecordset.actiongeocountrycode" : "US",           "eventrecordset.isrootevent" : 0,           "eventrecordset.issimulation" : false,           "eventrecordset.nummentions" : 10,           "eventrecordset.pmesiipttime" : 0.25,           "coalesceentity.objectkey" : "2c639634-7fd3-47bb-893d-120feb492ac9",           "eventrecordset.year" : 2018,           "eventrecordset.actiongeolocation" : "42.237300872802734, -71.53140258789062",           "eventrecordset.pmesiiptpolitical" : 0.5,           "eventrecordset.eventrootcode" : "04",           "eventrecordset.monthyear" : 201803,           "eventrecordset.actiongeotype" : 2,           "eventrecordset.datetime" : "2018-03-11T14:15:00.000Z",           "eventrecordset.numarticles" : 10,           "eventrecordset.datasource" : "GDELT",           "eventrecordset.eventbasecode" : "042",           "eventrecordset.dateadded" : 0         }       },       {         "_index" : "coalesce-oeevent",         "_type" : "recordset",         "_id" : "c831ec3b-2ab2-40b8-bbac-47ec627d3333", As you can see all data is there.   In regards to the nesting this is what shows up in the elasticsearch field configuration:

Use allShort names Use

Name

Type

Default Geometry

Stored

Analyzed

SRID

Date Format _id String

_index String

_type String

_score Float

_relative_score Float

_aggregation byte[]

coalesceentity.datecreated Date

date_optional_time coalesceentity.lastmodified Date

date_optional_time coalesceentity.name String

coalesceentity.objectkey String

coalesceentity.source String

coalesceentity.version String

eventrecordset.actiongeoadm1code String

x

eventrecordset.actiongeoadm1code.fields.keyword String

eventrecordset.actiongeoadm2code String

x

eventrecordset.actiongeoadm2code.fields.keyword String

eventrecordset.actiongeocountrycode String

x

eventrecordset.actiongeocountrycode.fields.keyword String

eventrecordset.actiongeofeatureid String

x

eventrecordset.actiongeofeatureid.fields.keyword String

eventrecordset.actiongeofullname String

x

eventrecordset.actiongeofullname.fields.keyword String

eventrecordset.actiongeolocation*

4326
eventrecordset.actiongeotype Integer

eventrecordset.avgtone Float

eventrecordset.datasource String

x

eventrecordset.datasource.fields.keyword String

eventrecordset.dateadded Integer

eventrecordset.datetime Date

date_optional_time eventrecordset.day Integer

eventrecordset.eventbasecode String

x

eventrecordset.eventbasecode.fields.keyword String

eventrecordset.eventcode String

x

eventrecordset.eventcode.fields.keyword String

eventrecordset.eventrootcode String

x

eventrecordset.eventrootcode.fields.keyword String

eventrecordset.fractiondate Float

eventrecordset.globaleventid Integer

eventrecordset.goldsteinscale Float

eventrecordset.isrootevent Integer

eventrecordset.issimulation Boolean

eventrecordset.monthyear Integer

eventrecordset.numarticles Integer

eventrecordset.nummentions Integer

eventrecordset.numsources Integer

eventrecordset.ontologyreference String

x

eventrecordset.ontologyreference.fields.keyword String

eventrecordset.pmesiipteconomic Float

eventrecordset.pmesiiptinformation Float

eventrecordset.pmesiiptinfrastructure Float

eventrecordset.pmesiiptmilitary Float

eventrecordset.pmesiiptphysicalenvironment Float

eventrecordset.pmesiiptpolitical Float

eventrecordset.pmesiiptsocial Float

eventrecordset.pmesiipttime Float

eventrecordset.quadclass Integer

eventrecordset.sourceurl String

x

eventrecordset.sourceurl.fields.keyword String

eventrecordset.tags String

x

eventrecordset.year Integer

Apply http://oeservices:8080/geoserver/web/wicket/page?24# So yes everyfield is a subfield.   When I do the GML or GEOJSON preview request only the non-subfields (_id,_type, _index, etc.) have values.

Does the plug-in not handle the nested subfields?

On 5/15/18 12:23 PM, sjudeng wrote:

Try adding the below lines to your logging file (see here https://github.com/ngageoint/elasticgeo/blob/master/gs-web-elasticsearch/doc/index.rst#configuring-logging)

|log4j.category.mil.nga.giat.data.elasticsearch=DEBUG log4j.category.mil.nga.giat.process.elasticsearch=DEBUG |

This should show the queries that are being sent to Elasticsearch and you can then execute them manually to see if there's any obvious issue with the response. What version of the plugin are you using? Are all the fields nested https://www.elastic.co/guide/en/elasticsearch/reference/5.4/nested.html?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ngageoint/elasticgeo/issues/69#issuecomment-389228244, or mute the thread https://github.com/notifications/unsubscribe-auth/ADslzGHMB9IkAE3EL0DWnZlwzROSju4yks5tywEPgaJpZM4T-x3C.

-- ========= mailto:dboyd@incadencecorp.com ============ David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ============== http://www.incadencecorp.com/ ============ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.

sjudeng commented 6 years ago

It should. Filtering isn't supported on nested geometries (it's okay for the object type). But this looks like an issue with feature reading not filtering. Can you post your Elasticsearch mapping (or a snippet including the geo and a couple of the other fields)?

bytemedwb commented 6 years ago

Sjudeng:

Below is the mapping. Attached is a dump of 5 records.

[root@node08 ~]# curl -k --cert "/etc/pki/java/temp.pem" -XGET "https://node08:9202/coalesce-oeevent?pretty" { "coalesce-oeevent" : { "aliases" : { }, "mappings" : { "recordset" : { "properties" : { "coalesceentity" : { "properties" : { "datecreated" : { "type" : "date" }, "lastmodified" : { "type" : "date" }, "name" : { "type" : "keyword" }, "objectkey" : { "type" : "keyword" }, "source" : { "type" : "keyword" }, "version" : { "type" : "keyword" } } }, "eventrecordset" : { "properties" : { "actiongeoadm1code" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "actiongeoadm2code" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "actiongeocountrycode" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "actiongeofeatureid" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "actiongeofullname" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "actiongeolocation" : { "type" : "geo_point" }, "actiongeotype" : { "type" : "integer" }, "avgtone" : { "type" : "float" }, "datasource" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "dateadded" : { "type" : "integer" }, "datetime" : { "type" : "date" }, "day" : { "type" : "integer" }, "eventbasecode" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "eventcode" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "eventrootcode" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "fractiondate" : { "type" : "float" }, "globaleventid" : { "type" : "integer" }, "goldsteinscale" : { "type" : "float" }, "isrootevent" : { "type" : "integer" }, "issimulation" : { "type" : "boolean" }, "monthyear" : { "type" : "integer" }, "numarticles" : { "type" : "integer" }, "nummentions" : { "type" : "integer" }, "numsources" : { "type" : "integer" }, "ontologyreference" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "pmesiipteconomic" : { "type" : "float" }, "pmesiiptinformation" : { "type" : "float" }, "pmesiiptinfrastructure" : { "type" : "float" }, "pmesiiptmilitary" : { "type" : "float" }, "pmesiiptphysicalenvironment" : { "type" : "float" }, "pmesiiptpolitical" : { "type" : "float" }, "pmesiiptsocial" : { "type" : "float" }, "pmesiipttime" : { "type" : "float" }, "quadclass" : { "type" : "integer" }, "sourceurl" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } }, "tags" : { "type" : "text" }, "year" : { "type" : "integer" } } } } } }, "settings" : { "index" : { "creation_date" : "1525982572184", "number_of_shards" : "5", "number_of_replicas" : "1", "uuid" : "t2Ww1t3eQbuyRca5fve8ZQ", "version" : { "created" : "5040099" }, "provided_name" : "coalesce-oeevent" } } } }

On 5/18/18 8:35 AM, sjudeng wrote:

It should. Filtering isn't supported on nested geometries (it's okay for the |object| type). But this looks like an issue with feature reading not filtering. Can you post your Elasticsearch mapping (or a snippet including the geo and a couple of the other fields)?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ngageoint/elasticgeo/issues/69#issuecomment-390192938, or mute the thread https://github.com/notifications/unsubscribe-auth/ADslzJumQpeGQvc07dQMJC1dvZYWxy56ks5tzsAUgaJpZM4T-x3C.

-- ========= mailto:dboyd@incadencecorp.com ============ David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ============== http://www.incadencecorp.com/ ============ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.

sjudeng commented 6 years ago

I have not been able to reproduce this yet. I took the mapping and sample document you provided and added them to ES 5.4.3.

curl -XPUT http://localhost:9200/coalesce-oeevent
curl -XPUT -H 'content-type: application/json' http://localhost:9200/coalesce-oeevent/_mapping/recordset -d @mapping.json
curl -XPOST -H 'content-type: application/json' http://localhost:9200/coalesce-oeevent/recordset -d @doc.json

I was then able to add the store and publish the layer in GeoServer. Fields in WFS (GML/GeoJSON) results were populated with expected non-null values.

curl "http://localhost:8080/geoserver/test/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=test:recordset&maxFeatures=50&outputFormat=json"

{
  "crs": {
    "type": "name", 
    "properties": {
      "name": "urn:ogc:def:crs:EPSG::4326"
    }
  }, 
  "totalFeatures": 1, 
  "type": "FeatureCollection", 
  "features": [
    {
      "geometry": {
        "type": "Point", 
        "coordinates": [
          -71.53140259, 
          42.23730087
        ]
      }, 
      "properties": {
        "coalesceentity.version": "0.2", 
        "eventrecordset.actiongeofeatureid": "MA", 
        "_score": 1, 
        "eventrecordset.ontologyreference.fields.keyword": "Event", 
        "eventrecordset.eventbasecode": "042", 
        "eventrecordset.dateadded": 0, 
        "eventrecordset.actiongeoadm2code.fields.keyword": "", 
        "eventrecordset.actiongeoadm1code.fields.keyword": "USMA", 
        "_aggregation": null, 
        "coalesceentity.source": "G2Core", 
        "eventrecordset.pmesiipttime": 0.25, 
        "eventrecordset.eventrootcode": "04", 
        "_relative_score": 1, 
        "coalesceentity.name": "OEEvent", 
        "eventrecordset.actiongeocountrycode.fields.keyword": "US", 
        "eventrecordset.pmesiiptmilitary": 0.25, 
        "eventrecordset.datasource.fields.keyword": "GDELT", 
        "eventrecordset.day": 20180311, 
        "eventrecordset.eventcode": "042", 
        "eventrecordset.pmesiiptinfrastructure": 0.25, 
        "eventrecordset.avgtone": -4.989154, 
        "eventrecordset.pmesiiptpolitical": 0.5, 
        "eventrecordset.actiongeofeatureid.fields.keyword": "MA", 
        "eventrecordset.numarticles": 10, 
        "eventrecordset.numsources": 1, 
        "eventrecordset.actiongeofullname.fields.keyword": "Massachusetts, United States", 
        "_type": "recordset", 
        "eventrecordset.issimulation": false, 
        "eventrecordset.nummentions": 10, 
        "eventrecordset.tags": null, 
        "eventrecordset.globaleventid": 737769639, 
        "eventrecordset.ontologyreference": "Event", 
        "eventrecordset.datasource": "GDELT", 
        "eventrecordset.actiongeofullname": "Massachusetts, United States", 
        "eventrecordset.monthyear": 201803, 
        "coalesceentity.objectkey": "2c639634-7fd3-47bb-893d-120feb492ac9", 
        "eventrecordset.eventrootcode.fields.keyword": "04", 
        "eventrecordset.sourceurl.fields.keyword": "http://nhpr.org/post/winter-storm-watch-tuesday-nhs-town-meeting-day", 
        "coalesceentity.lastmodified": "2018-03-11T15:11:33.996Z", 
        "eventrecordset.sourceurl": "http://nhpr.org/post/winter-storm-watch-tuesday-nhs-town-meeting-day", 
        "eventrecordset.actiongeotype": 2, 
        "eventrecordset.actiongeocountrycode": "US", 
        "eventrecordset.year": 2018, 
        "eventrecordset.quadclass": 1, 
        "eventrecordset.datetime": "2018-03-11T14:15:00Z", 
        "eventrecordset.pmesiiptinformation": 0.5, 
        "eventrecordset.actiongeoadm2code": "", 
        "eventrecordset.actiongeoadm1code": "USMA", 
        "eventrecordset.isrootevent": 0, 
        "eventrecordset.pmesiipteconomic": 0.25, 
        "_index": "coalesce-oeevent", 
        "eventrecordset.pmesiiptphysicalenvironment": 0.5, 
        "coalesceentity.datecreated": "2018-03-11T15:11:33.985Z", 
        "eventrecordset.goldsteinscale": 1.9, 
        "eventrecordset.eventcode.fields.keyword": "042", 
        "eventrecordset.eventbasecode.fields.keyword": "042", 
        "eventrecordset.pmesiiptsocial": 0.5, 
        "_id": "AWOfnRqCqTF0HjaY37Wm", 
        "eventrecordset.fractiondate": 2018.1945
      }, 
      "type": "Feature", 
      "id": "recordset.AWOfnRqCqTF0HjaY37Wm", 
      "geometry_name": "eventrecordset.actiongeolocation"
    }
  ]
}

Compute from bounds also worked as expected when adding the feature type. Note in practice with larger number of documents it's recommended to manually enter the layer bounds to avoid expensive computation, though it doesn't explain the issue with null values in your features.

bytemedwb commented 6 years ago

Sjudeng:

Thanks. This is puzzling. Will look into what is happening.

On 5/26/18 11:43 PM, sjudeng wrote:

I have not been able to reproduce this yet. I took the mapping https://github.com/ngageoint/elasticgeo/files/2042248/mapping.txt and sample document https://github.com/ngageoint/elasticgeo/files/2042249/doc.txt you provided and added them to ES 5.4.3.

curl -XPUT http://localhost:9200/coalesce-oeevent curl -XPUT -H'content-type: application/json' http://localhost:9200/coalesce-oeevent/_mapping/recordset -d @mapping.json curl -XPOST -H'content-type: application/json' http://localhost:9200/coalesce-oeevent/recordset -d @doc.json

I was then able to add the store and publish the layer in GeoServer. Fields in WFS (GML/GeoJSON) results were populated with expected non-null values.

curl"http://localhost:8080/geoserver/test/ows?service=WFS&version=1.0.0&request=GetFeature&typeName=test:recordset&maxFeatures=50&outputFormat=json" { "crs": { "type":"name", "properties": { "name":"urn:ogc:def:crs:EPSG::4326" } }, "totalFeatures":1, "type":"FeatureCollection", "features": [ { "geometry": { "type":"Point", "coordinates": [ -71.53140259, 42.23730087 ] }, "properties": { "coalesceentity.version":"0.2", "eventrecordset.actiongeofeatureid":"MA", "_score":1, "eventrecordset.ontologyreference.fields.keyword":"Event", "eventrecordset.eventbasecode":"042", "eventrecordset.dateadded":0, "eventrecordset.actiongeoadm2code.fields.keyword":"", "eventrecordset.actiongeoadm1code.fields.keyword":"USMA", "_aggregation":null, "coalesceentity.source":"G2Core", "eventrecordset.pmesiipttime":0.25, "eventrecordset.eventrootcode":"04", "_relative_score":1, "coalesceentity.name":"OEEvent", "eventrecordset.actiongeocountrycode.fields.keyword":"US", "eventrecordset.pmesiiptmilitary":0.25, "eventrecordset.datasource.fields.keyword":"GDELT", "eventrecordset.day":20180311, "eventrecordset.eventcode":"042", "eventrecordset.pmesiiptinfrastructure":0.25, "eventrecordset.avgtone":-4.989154, "eventrecordset.pmesiiptpolitical":0.5, "eventrecordset.actiongeofeatureid.fields.keyword":"MA", "eventrecordset.numarticles":10, "eventrecordset.numsources":1, "eventrecordset.actiongeofullname.fields.keyword":"Massachusetts, United States", "_type":"recordset", "eventrecordset.issimulation":false, "eventrecordset.nummentions":10, "eventrecordset.tags":null, "eventrecordset.globaleventid":737769639, "eventrecordset.ontologyreference":"Event", "eventrecordset.datasource":"GDELT", "eventrecordset.actiongeofullname":"Massachusetts, United States", "eventrecordset.monthyear":201803, "coalesceentity.objectkey":"2c639634-7fd3-47bb-893d-120feb492ac9", "eventrecordset.eventrootcode.fields.keyword":"04", "eventrecordset.sourceurl.fields.keyword":"http://nhpr.org/post/winter-storm-watch-tuesday-nhs-town-meeting-day", "coalesceentity.lastmodified":"2018-03-11T15:11:33.996Z", "eventrecordset.sourceurl":"http://nhpr.org/post/winter-storm-watch-tuesday-nhs-town-meeting-day", "eventrecordset.actiongeotype":2, "eventrecordset.actiongeocountrycode":"US", "eventrecordset.year":2018, "eventrecordset.quadclass":1, "eventrecordset.datetime":"2018-03-11T14:15:00Z", "eventrecordset.pmesiiptinformation":0.5, "eventrecordset.actiongeoadm2code":"", "eventrecordset.actiongeoadm1code":"USMA", "eventrecordset.isrootevent":0, "eventrecordset.pmesiipteconomic":0.25, "_index":"coalesce-oeevent", "eventrecordset.pmesiiptphysicalenvironment":0.5, "coalesceentity.datecreated":"2018-03-11T15:11:33.985Z", "eventrecordset.goldsteinscale":1.9, "eventrecordset.eventcode.fields.keyword":"042", "eventrecordset.eventbasecode.fields.keyword":"042", "eventrecordset.pmesiiptsocial":0.5, "_id":"AWOfnRqCqTF0HjaY37Wm", "eventrecordset.fractiondate":2018.1945 }, "type":"Feature", "id":"recordset.AWOfnRqCqTF0HjaY37Wm", "geometry_name":"eventrecordset.actiongeolocation" } ] }

Compute from bounds also worked as expected when adding the feature type. Note in practice with larger number of documents it's recommended to manually enter the layer bounds to avoid expensive computation, though it doesn't explain the issue with null values in your features.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ngageoint/elasticgeo/issues/69#issuecomment-392303278, or mute the thread https://github.com/notifications/unsubscribe-auth/ADslzKHOr1MfNETxKmntWWqrcD9y5gysks5t2iDYgaJpZM4T-x3C.

-- ========= mailto:dboyd@incadencecorp.com ============ David W. Boyd VP, Data Solutions 10432 Balls Ford, Suite 240 Manassas, VA 20109 office: +1-703-552-2862 cell: +1-703-402-7908 ============== http://www.incadencecorp.com/ ============ ISO/IEC JTC1 WG9, editor ISO/IEC 20547 Big Data Reference Architecture Chair ANSI/INCITS TC Big Data Co-chair NIST Big Data Public Working Group Reference Architecture First Robotic Mentor - FRC, FTC - www.iliterobotics.org Board Member- USSTEM Foundation - www.usstem.org

The information contained in this message may be privileged and/or confidential and protected from disclosure. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by replying to this message and deleting the material from any computer.

dclemenzi commented 6 years ago

Here are some additional details. Placing a break point on line 313 in ElasticParserUtil here is the content of the source mapping:

source = {LinkedHashMap@7557}  size = 10
 0 = {LinkedHashMap$Entry@7956} "test1.int" -> "10"
 1 = {LinkedHashMap$Entry@7957} "coalesceentity.name" -> "UNIT_TEST"
 2 = {LinkedHashMap$Entry@7958} "test1.geo" -> "-3.18, 51.4347"
 3 = {LinkedHashMap$Entry@7959} "coalesceentity.datecreated" -> "2018-06-07T14:47:57.343Z"
 4 = {LinkedHashMap$Entry@7960} "coalesceentity.source" -> "DSS"
 5 = {LinkedHashMap$Entry@7961} "test1.string" -> "EUROPE"
 6 = {LinkedHashMap$Entry@7962} "test1.date" -> "2018-06-07T14:47:57.374Z"
 7 = {LinkedHashMap$Entry@7963} "coalesceentity.lastmodified" -> "2018-06-07T14:47:57.375Z"
 8 = {LinkedHashMap$Entry@7964} "coalesceentity.version" -> "1.0"
 9 = {LinkedHashMap$Entry@7965} "coalesceentity.objectkey" -> "b4cbb5fa-7106-4b75-a2bc-791f88cc8373"

Line 310 splits the name being passed in so on line 313 its looking for the key coalesceentity instead of coalesceentity.name which results in a null value. If line 313 could pass in the name unmodified the code would work. I don't understand ElasticSearch well enough to know why its splitting the name in the first place. Any guidance would be appreciate.

I tried pulling the source and modifing it but that resulted in a non-shaded jar. How was the 2.13.0 version compiled and deployed?

dclemenzi commented 6 years ago

Don't know if this is a bad approach or not but this code change resolves the issue:

    public List<Object> readField(Map<String, Object> source, String name) {
        final List<String> keys = Arrays.asList(name.split("\\."));
        List<Object> values = new ArrayList<>();
        if (!keys.isEmpty()) {
            readField(source.get(keys.get(0)), keys.subList(1, keys.size()), values);
        }
        final List<Object> result;
        if (!values.isEmpty()) {
            result = values;
        } else {
            result = null;
        }
        return result;
    }

To:

public List<Object> readField(Map<String, Object> source, String name) {
        final List<String> keys = Arrays.asList(name.split("\\."));
        List<Object> values = new ArrayList<>();
        if (!keys.isEmpty()) {

            Object entry = source.get(keys.get(0));

            if (entry == null)
            {
                entry = source.get(name);
            }

            readField(entry, keys.subList(1, keys.size()), values);
        }
        final List<Object> result;
        if (!values.isEmpty()) {
            result = values;
        } else {
            result = null;
        }
        return result;
    }

sjudeng commented 6 years ago

The change looks reasonable, can you add a test and open a PR with the update? Looks like you figured it out but when building locally you'll find the shaded jar in gs-web-elasticsearch/target/elasticgeo-2.13-SNAPSHOT.jar.

dclemenzi commented 6 years ago

Getting a 403 when attempting to push to a branch. Not sure what I am missing.

sjudeng commented 6 years ago

Pushing to your fork or this repo? You'd only be able to push to your fork and then you can open a PR from there. If you haven't already click Fork above and then in your local folder do a git remote add dclemenzi https://github.com/dclemenzi/elasticgeo.git. Then you should be able to do a git push dclemenzi your-branch.

sjudeng commented 6 years ago

Fixed by #71 (thanks @dclemenzi)

ngageoint / elasticgeo

Invalid bounds from data and all fields beside elastic generated (_type) are null #69