search

orientechnologies / orientdb

OrientDB is the most versatile DBMS supporting Graph, Document, Reactive, Full-Text and Geospatial models in one Multi-Model product. OrientDB can run distributed (Multi-Master), supports SQL, ACID Transactions, Full-Text indexing and Reactive Queries.
https://orientdb.dev
Apache License 2.0
4.73k stars 871 forks source link

Error when creating an Edge to an existing Vertex #5163

Closed robfrank closed 3 years ago

robfrank commented 8 years ago

From @padutilleul on August 18, 2015 15:17

Tested on OrientDB 2.1.0 GA Community Edition

To replicate, launch an instance of orientDB server, create a new database, and run this oetl script

Here is the data :

[
  {
    "name": "Parent",
    "idf": 1
  },
  {
    "name": "Child",
    "idf": 2,
    "parent": 1
  }
]

And the oetl script : 

{
  "config": {
    "log": "DEBUG"
  },
  "source" : {
    "file": { "path": "<path to data file>", "lock" : true }
  },
  "extractor" : {
    "json": {}
  },
  "transformers" : [
    { "merge": { "joinFieldName":"idf", "lookup":"V.idf" } },
    { "vertex": { "class": "V"} },
    { "edge": {
      "class": "E",
      "joinFieldName": "parent",
      "lookup": "V.idf",
      "unresolvedLinkAction": "CREATE",
      "if": "$input.parent IS NOT NULL"
    } }
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "remote:localhost/<database name>",
      "dbUser": "root",
      "dbPassword": "***",
      "dbType": "graph",
      "indexes": [{"class":"V", "fields":["idf:integer"], "type":"UNIQUE" }]
    }
  }
}

And here is the output : 

OrientDB etl v.2.1.0 (build @BUILD@) www.orientdb.com
BEGIN ETL PROCESSOR
[file] DEBUG Reading from file /Users/pa/Documents/orientDB/importETL/data2.json
[orientdb] DEBUG - OrientDBLoader: created property 'V.idf' of type: integer
[orientdb] DEBUG - OrientDocumentLoader: created index 'V.idf' type 'UNIQUE' against Class 'V', fields [idf:integer]
[0:merge] DEBUG Transformer input: {name:Parent,idf:1}
[0:merge] DEBUG joinValue=1, lookupResult=null
[0:merge] DEBUG Transformer output: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer input: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer output: v[#9:0]
[0:edge] DEBUG Transformer input: v[#9:0]
[0:edge] DEBUG Evaluating conditional expression if=Parsed: ($input.field(parent) IS '_NOT_NULL_')...
[0:edge] DEBUG Transformer output (same as input): v[#9:0]
[1:merge] DEBUG Transformer input: {name:Child,idf:2,parent:1}
[1:merge] DEBUG joinValue=2, lookupResult=null
[1:merge] DEBUG Transformer output: {name:Child,idf:2,parent:1}
[1:vertex] DEBUG Transformer input: {name:Child,idf:2,parent:1}
[1:vertex] DEBUG Transformer output: v[#9:1]
[1:edge] DEBUG Transformer input: v[#9:1]
[1:edge] DEBUG Evaluating conditional expression if=Parsed: ($input.field(parent) IS '_NOT_NULL_')...
[1:edge] DEBUG joinCurrentValue=1, lookupResult=V#9:0{name:Parent,idf:1} v1
Error in Pipeline execution: java.lang.IllegalArgumentException: destination vertex is null
java.lang.IllegalArgumentException: destination vertex is null
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:888)
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:832)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.createEdge(OEdgeTransformer.java:184)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.executeTransform(OEdgeTransformer.java:117)
    at com.orientechnologies.orient.etl.transformer.OAbstractTransformer.transform(OAbstractTransformer.java:37)
    at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:114)
    at com.orientechnologies.orient.etl.OETLProcessor.executeSequentially(OETLProcessor.java:487)
    at com.orientechnologies.orient.etl.OETLProcessor.execute(OETLProcessor.java:291)
    at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:161)
ETL process halted: com.orientechnologies.orient.etl.OETLProcessHaltedException: java.lang.IllegalArgumentException: destination vertex is null

Copied from original issue: orientechnologies/orientdb-etl#77

robfrank commented 8 years ago

From @lvca on August 19, 2015 16:34

The problem is that the IF block must be outside the inner edge block. Try this:

{ "edge": {
      "class": "E",
      "joinFieldName": "parent",
      "lookup": "V.idf",
      "unresolvedLinkAction": "CREATE"
    }, 
    "if": "$input.parent IS NOT NULL" }
robfrank commented 8 years ago

From @padutilleul on August 20, 2015 10:46

I tried your change : 

{
  "config": {
    "log": "DEBUG"
  },
  "source" : {
    "file": { "path": "<path to data file>", "lock" : true }
  },
  "extractor" : {
    "json": {}
  },
  "transformers" : [
    { "merge": { "joinFieldName":"idf", "lookup":"V.idf" } },
    { "vertex": { "class": "V"} },
    { "edge": {
        "class": "E",
        "joinFieldName": "parent",
        "lookup": "V.idf",
        "unresolvedLinkAction": "CREATE"
      },
      "if": "$input.parent IS NOT NULL" 
    }
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "remote:localhost/<database name>",
      "dbUser": "root",
      "dbPassword": "***",
      "dbType": "graph",
      "indexes": [{"class":"V", "fields":["idf:integer"], "type":"UNIQUE" }]
    }
  }
}

Here is the output : 

OrientDB etl v.2.1.0 (build @BUILD@) www.orientdb.com
BEGIN ETL PROCESSOR
[file] DEBUG Reading from file /Users/pa/Documents/orientDB/importETL/data2.json
[orientdb] DEBUG - OrientDBLoader: created property 'V.idf' of type: integer
[orientdb] DEBUG - OrientDocumentLoader: created index 'V.idf' type 'UNIQUE' against Class 'V', fields [idf:integer]
[0:merge] DEBUG Transformer input: {name:Parent,idf:1}
[0:merge] DEBUG joinValue=1, lookupResult=null
[0:merge] DEBUG Transformer output: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer input: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer output: v[#9:0]
[0:edge] DEBUG Transformer input: v[#9:0]
[0:edge] DEBUG joinCurrentValue=null, lookupResult=null
Error in Pipeline execution: java.lang.IllegalArgumentException: Property value can not be null
java.lang.IllegalArgumentException: Property value can not be null
    at com.tinkerpop.blueprints.util.ExceptionFactory.propertyValueCanNotBeNull(ExceptionFactory.java:60)
    at com.tinkerpop.blueprints.impls.orient.OrientElement.validateProperty(OrientElement.java:575)
    at com.tinkerpop.blueprints.impls.orient.OrientElement.setProperty(OrientElement.java:195)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.createEdge(OEdgeTransformer.java:136)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.executeTransform(OEdgeTransformer.java:117)
    at com.orientechnologies.orient.etl.transformer.OAbstractTransformer.transform(OAbstractTransformer.java:37)
    at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:114)
    at com.orientechnologies.orient.etl.OETLProcessor.executeSequentially(OETLProcessor.java:487)
    at com.orientechnologies.orient.etl.OETLProcessor.execute(OETLProcessor.java:291)
    at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:161)
ETL process halted: com.orientechnologies.orient.etl.OETLProcessHaltedException: java.lang.IllegalArgumentException: Property value can not be null

So it seems the if condition is not tested if you place it outside the inner edge block

robfrank commented 8 years ago

From @lvca on August 20, 2015 16:29

I have created this test case and passes:

  @Test
  public void testCreateTargetVertexIfNotExists() {
    process("{source: { content: { value: 'name,idf,parent\nParent,1,\nChild,2,1' } }, extractor : { row: {} },"
        + " transformers: [{csv: {}}, {merge: { joinFieldName:'idf', lookup:'V.idf'}}, {vertex: {class:'V'}},"+
              "{edge:{ class: 'E', joinFieldName: 'parent', lookup: 'V.idf', unresolvedLinkAction: 'CREATE' }, if: '$input.parent IS NOT NULL'}"
        + "], loader: { orientdb: { dbURL: 'memory:ETLBaseTest', dbType:'graph', useLightweightEdges:false } } }");

    assertEquals(2, graph.countVertices("V"));
  }

Please could you try using 2.1.1-SNAPSHOT?

robfrank commented 8 years ago

From @padutilleul on August 21, 2015 9:21

I installed 2.1.1-SNAPSHOT, but there is no oetl.sh script in the bin directory.

I am currently trying to compile it from your oreintdb-etl git repository, but I am running into problems with maven so it is going to take a little while.

On a side note, when I run my script (or your test case version), the 2 vertices are created, but the script crashes when it tries to create the egde between them. I am not familiar with your test case framework, but maybe it's why your test case passes as it only check for the vertices and not the edge ?

robfrank commented 8 years ago

From @padutilleul on August 21, 2015 13:14

Ok, I managed to compile orientdb-etl.2.1.1-SNAPSHOT.jar from your git repository (branch 2.1.x).

ran the previous script, and here is the output : 

OrientDB etl v.2.1.1-SNAPSHOT (build @BUILD@) www.orientdb.com
BEGIN ETL PROCESSOR
[file] DEBUG Reading from file /Users/pa/Documents/orientDB/importETL/data2.json
[orientdb] DEBUG - OrientDBLoader: created property 'V.idf' of type: integer
[orientdb] DEBUG - OrientDocumentLoader: created index 'V.idf' type 'UNIQUE' against Class 'V', fields [idf:integer]
[0:merge] DEBUG Transformer input: {name:Parent,idf:1}
[0:merge] DEBUG joinValue=1, lookupResult=null
[0:merge] DEBUG Transformer output: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer input: {name:Parent,idf:1}
[0:vertex] DEBUG Transformer output: v[#9:0]
[0:edge] DEBUG Transformer input: v[#9:0]
[0:edge] DEBUG joinCurrentValue=null, lookupResult=null
[0:edge] DEBUG Transformer output: v[#9:0]
[1:merge] DEBUG Transformer input: {name:Child,idf:2,parent:1}
[1:merge] DEBUG joinValue=2, lookupResult=null
[1:merge] DEBUG Transformer output: {name:Child,idf:2,parent:1}
[1:vertex] DEBUG Transformer input: {name:Child,idf:2,parent:1}
[1:vertex] DEBUG Transformer output: v[#9:1]
[1:edge] DEBUG Transformer input: v[#9:1]
[1:edge] DEBUG joinCurrentValue=1, lookupResult=V#9:0{name:Parent,idf:1} v1
Error in Pipeline execution: java.lang.IllegalArgumentException: destination vertex is null
java.lang.IllegalArgumentException: destination vertex is null
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:888)
    at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:832)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.createEdge(OEdgeTransformer.java:188)
    at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.executeTransform(OEdgeTransformer.java:117)
    at com.orientechnologies.orient.etl.transformer.OAbstractTransformer.transform(OAbstractTransformer.java:37)
    at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:114)
    at com.orientechnologies.orient.etl.OETLProcessor.executeSequentially(OETLProcessor.java:487)
    at com.orientechnologies.orient.etl.OETLProcessor.execute(OETLProcessor.java:291)
    at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:161)
ETL process halted: com.orientechnologies.orient.etl.OETLProcessHaltedException: java.lang.IllegalArgumentException: destination vertex is null

I also tried to remove different parts of the json script to narrow it down :

So the simplest way I found to trigger this bug is : 

{
  "config": {
    "log": "DEBUG"
  },
  "source": {
    "content": { "value": "name,idf,parent\nParent,1,\nChild,2,1" }
  },
  "extractor" : {
    "row": {}
  },
  "transformers" : [
    {"csv": {}},
    { "vertex": { "class": "V"} },
    { "edge": {
        "class": "E",
        "joinFieldName": "parent",
        "lookup": "V.idf"
    }}
  ],
  "loader" : {
    "orientdb": {
      "dbURL": "remote:localhost/<database name>",
      "dbUser": "root",
      "dbPassword": "***",
      "dbType": "graph",
      "indexes": [{"class":"V", "fields":["idf:integer"], "type":"UNIQUE" }]
    }
  }
}
robfrank commented 8 years ago

From @lvca on August 21, 2015 14:11

Thanks, I'm on it ;-)

robfrank commented 8 years ago

From @padutilleul on September 2, 2015 9:11

May I help in any way ?

Do not hesitate to ask if you want me to run some more tests.

robfrank commented 8 years ago

From @fluxlife on September 18, 2015 10:42

I am also having similar issues. Running Orient 2.1.2 Community.

I have successfully loaded one class called 'Account' Then when I try to import User(s) with this config:

{
  "config": {
    "log": "debug"
  },
  "extractor": {
    "jdbc": {
      "driver": "org.postgresql.Driver",
      "url": "jdbc:...",
      "userName": "...",
      "userPassword": "...",
      "query": "select * from ..."
    }
  },
  "transformers": [
    {
      "vertex": {
        "class": "User"
      }
    },
    {
      "edge": {
        "class": "is_user",
        "joinFieldName": "account_id",
        "lookup": "Account.id",
        "direction": "out",
        "unresolvedLinkAction": "CREATE"
      }
    }
  ],
  "loader": {
    "orientdb": {
      "dbURL": "plocal:/etc/orientdb-community-2.1.2/databases/viz",
      "dbType": "graph",
      "dbAutoCreate": true,
      "classes": [
        {
          "name": "User",
          "extends": "TV"
        },
        {
          "name": "Account",
          "extends": "TV"
        },
        {
          "name": "is_user",
          "extends": "TE"
        }
      ]
    }
  }
}

I get

OrientDB etl v.2.1.2 (build @BUILD@) www.orientdb.com
BEGIN ETL PROCESSOR
[orientdb] DEBUG orientdb: found 1 vertices in class 'null'
[orientdb] DEBUG orientdb: found 3192 vertices in class 'null'
[orientdb] DEBUG orientdb: found 0 vertices in class 'null'
[1:vertex] DEBUG Transformer input: {id:24,account_id:2,...}
+ extracted 2 records (0 records/sec) - 2 records -> loaded 0 vertices (0 vertices/sec) Total time: 1000ms [0 warnings, 0 errors]
[1:vertex] DEBUG Transformer output: v(User)[#16:1]
[1:edge] DEBUG Transformer input: v(User)[#16:1]
[1:edge] DEBUG joinCurrentValue=2, lookupResult=Account#14:2609{id:2,....} v2
Error in Pipeline execution: java.lang.IllegalArgumentException: destination vertex is null
java.lang.IllegalArgumentException: destination vertex is null
        at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:888)
        at com.tinkerpop.blueprints.impls.orient.OrientVertex.addEdge(OrientVertex.java:832)
        at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.createEdge(OEdgeTransformer.java:188)
        at com.orientechnologies.orient.etl.transformer.OEdgeTransformer.executeTransform(OEdgeTransformer.java:117)
        at com.orientechnologies.orient.etl.transformer.OAbstractTransformer.transform(OAbstractTransformer.java:37)
        at com.orientechnologies.orient.etl.OETLPipeline.execute(OETLPipeline.java:114)
        at com.orientechnologies.orient.etl.OETLProcessor.executeSequentially(OETLProcessor.java:487)
        at com.orientechnologies.orient.etl.OETLProcessor.execute(OETLProcessor.java:291)
        at com.orientechnologies.orient.etl.OETLProcessor.main(OETLProcessor.java:161)
ETL process halted: com.orientechnologies.orient.etl.OETLProcessHaltedException: java.lang.IllegalArgumentException: destination vertex is null

Please note I have modified some of the config and output for security and brevity reasons. When I was trying to add 'indexes' I was getting other errors, so I took that line out.