search

graphaware / neo4j-to-elasticsearch

GraphAware Framework Module for Integrating Neo4j with Elasticsearch
261 stars 57 forks source link

Error while creating json. Missing keyProperty #143

Closed oleksii-novikov closed 5 years ago

oleksii-novikov commented 5 years ago

During the indexing process we see a lot of errors saying next:

com.graphaware.module.es.mapping.json.DocumentRepresentationException: Error while creating json. Missing keyProperty
   at com.graphaware.module.es.mapping.json.GraphDocumentMapper.getKeyProperty(GraphDocumentMapper.java:137)

I believe that happens because some of the properties could be missed. Is it possible to skip non-existed properties and not throw errors if a property is missed? Setting exclude_empty_properties to true doesn't help, it seems that it influences only on putting data to ES, and not on retrieving. Here is my mapping.json file

{
  "defaults": {
    "key_property": "uuid",
    "nodes_index": "default-index-node",
    "relationships_index": "default-index-relationship",
    "include_remaining_properties": false,
    "exclude_empty_properties": true
  },
  "node_mappings": [
    {
      "condition": "hasLabel('Paper')",
      "index": "nodes-papers",
      "type": "papers",
      "properties": {
        "paperId": "getProperty('paperId')",
        "doi": "getProperty('doi')",
        "docType": "getProperty('docType')",
        "paperTitle": "getProperty('paperTitle')"
      }
    }
  ],
  "relationship_mappings": []
}

UPD: Even if I set in properties to index only one property, which is 100% exists in all nodes, I get the same errors.

We are using:

Neo4j-3.4.8 ElasticSearch - 6.3 Graphaware Plugin :

oleksii-novikov commented 5 years ago

Errors happened because old nodes didn't have uuid property and that property was specified as a key_property. To handle this:

dbms.unmanaged_extension_classes=com.graphaware.server=/graphaware
com.graphaware.runtime.enabled=true
com.graphaware.module.UIDM.1=com.graphaware.module.uuid.UuidBootstrapper
# this property will be added to all nodes
com.graphaware.module.UIDM.uuidProperty=uuid
com.graphaware.module.UIDM.node=hasLabel('Paper')
com.graphaware.module.UIDM.uuidIndex=uuidIndex
#################################################
# Set this to high value if you want to process all uuid first time
#################################################
com.graphaware.module.UIDM.initializeUntil=9900000000000000

After restart, uuid will be added to all nodes and then, when ES indexes will be created, there will be no errors Missing keyProperty