search

neo4j / graph-data-science

Source code for the Neo4j Graph Data Science library of graph algorithms.
https://neo4j.com/docs/graph-data-science/current/
Other
623 stars 160 forks source link

Error while using `gds.alpha.ml.ann.stream` #30

Closed karrtikiyer closed 4 years ago

karrtikiyer commented 4 years ago

Failed to invoke procedure gds.alpha.ml.ann.stream: Caused by: java.lang.IllegalMonitorStateException Cypher query:

MATCH (p:PERSON)-[:SKILLED_IN]->(SKILL)
 WITH {item:id(p), categories: collect(id(SKILL))} AS userData
 WITH collect(userData) AS data
 CALL gds.alpha.ml.ann.stream({
   nodeProjection: '*',
   relationshipProjection: '*',
   data: data,
   algorithm: 'jaccard',
   similarityCutoff: 0.1,
   concurrency: 1
 })
 YIELD item1, item2, similarity
 return gds.util.asNode(item1).name AS from, gds.util.asNode(item2).name AS to, similarity
 ORDER BY from

Error from log:

2020-04-17 13:26:34.841+0000 WARN  Computation failed
java.lang.IllegalMonitorStateException
    at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:154)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1265)
    at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:462)
    at org.neo4j.graphalgo.core.loading.ThreadLocalRelationshipsBuilder.unlock(ThreadLocalRelationshipsBuilder.java:75)
    at org.neo4j.graphalgo.core.loading.AdjacencyBuilder$CompressingPagedAdjacency.addAll(AdjacencyBuilder.java:225)
    at org.neo4j.graphalgo.core.loading.RelationshipImporter.importRelationships(RelationshipImporter.java:206)
    at org.neo4j.graphalgo.core.loading.RelationshipImporter.importNatural(RelationshipImporter.java:97)
    at org.neo4j.graphalgo.core.loading.GraphGenerator$RelImporter.flushBuffer(GraphGenerator.java:265)
    at org.neo4j.graphalgo.core.loading.GraphGenerator$RelImporter.buildGraph(GraphGenerator.java:238)
    at org.neo4j.graphalgo.impl.similarity.ApproxNearestNeighborsAlgorithm$RelationshipImporter.buildGraphStore(ApproxNearestNeighborsAlgorithm.java:554)
    at org.neo4j.graphalgo.impl.similarity.ApproxNearestNeighborsAlgorithm.similarityStream(ApproxNearestNeighborsAlgorithm.java:150)
    at org.neo4j.graphalgo.impl.similarity.SimilarityAlgorithm.generateWeightedStream(SimilarityAlgorithm.java:119)
    at org.neo4j.graphalgo.impl.similarity.SimilarityAlgorithm.compute(SimilarityAlgorithm.java:80)
    at org.neo4j.graphalgo.impl.similarity.SimilarityAlgorithm.compute(SimilarityAlgorithm.java:33)
    at org.neo4j.graphalgo.AlgoBaseProc.lambda$compute$1(AlgoBaseProc.java:297)
    at org.neo4j.graphalgo.BaseProc.runWithExceptionLogging(BaseProc.java:82)
    at org.neo4j.graphalgo.AlgoBaseProc.compute(AlgoBaseProc.java:293)
    at org.neo4j.graphalgo.AlgoBaseProc.compute(AlgoBaseProc.java:258)
    at org.neo4j.graphalgo.similarity.SimilarityProc.stream(SimilarityProc.java:53)
    at org.neo4j.graphalgo.similarity.ApproxNearestNeighborsProc.annStream(ApproxNearestNeighborsProc.java:75)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.neo4j.kernel.impl.proc.ReflectiveProcedureCompiler$ReflectiveProcedure.apply(ReflectiveProcedureCompiler.java:664)
    at org.neo4j.kernel.impl.proc.ProcedureRegistry.callProcedure(ProcedureRegistry.java:219)
    at org.neo4j.kernel.impl.proc.Procedures.callProcedure(Procedures.java:290)
    at org.neo4j.kernel.impl.newapi.AllStoreHolder.callProcedure(AllStoreHolder.java:1098)
    at org.neo4j.kernel.impl.newapi.AllStoreHolder.procedureCallRead(AllStoreHolder.java:863)
    at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$$anonfun$2.apply(CallSupport.scala:62)
    at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$$anonfun$2.apply(CallSupport.scala:62)
    at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$.callProcedure(CallSupport.scala:167)
    at org.neo4j.cypher.internal.runtime.interpreted.CallSupport$.callReadOnlyProcedure(CallSupport.scala:64)
    at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext.callReadOnlyProcedure(TransactionBoundQueryContext.scala:974)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$$anonfun$callReadOnlyProcedure$1.apply(ExceptionTranslatingQueryContext.scala:166)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext$$anonfun$callReadOnlyProcedure$1.apply(ExceptionTranslatingQueryContext.scala:166)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslationSupport$class.translateException(ExceptionTranslationSupport.scala:33)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext.translateException(ExceptionTranslatingQueryContext.scala:42)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslationSupport$class.translateIterator(ExceptionTranslationSupport.scala:47)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext.translateIterator(ExceptionTranslatingQueryContext.scala:42)
    at org.neo4j.cypher.internal.compatibility.v3_5.ExceptionTranslatingQueryContext.callReadOnlyProcedure(ExceptionTranslatingQueryContext.scala:166)
    at org.neo4j.cypher.internal.runtime.LazyReadOnlyCallMode.callProcedure(ProcedureCallMode.scala:48)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.ProcedureCallPipe.org$neo4j$cypher$internal$runtime$interpreted$pipes$ProcedureCallPipe$$call(ProcedureCallPipe.scala:90)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.ProcedureCallPipe$$anon$$$$e6882d5096c8b8441ac36d3c0cc459e$$$$ateResultsByAppending$1.apply(ProcedureCallPipe.scala:73)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.ProcedureCallPipe$$anon$$$$e6882d5096c8b8441ac36d3c0cc459e$$$$ateResultsByAppending$1.apply(ProcedureCallPipe.scala:71)
    at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:435)
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:441)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
    at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
    at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
    at scala.collection.AbstractIterator.to(Iterator.scala:1334)
    at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
    at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1334)
    at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
    at scala.collection.AbstractIterator.toArray(Iterator.scala:1334)
    at org.neo4j.cypher.internal.runtime.slotted.pipes.SortSlottedPipe.internalCreateResults(SortSlottedPipe.scala:29)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:76)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:72)
    at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:72)
    at org.neo4j.cypher.internal.compatibility.v3_5.runtime.executionplan.BaseExecutionResultBuilderFactory$BaseExecutionWorkflowBuilder.build(DefaultExecutionResultBuilderFactory.scala:61)
    at org.neo4j.cypher.internal.compatibility.InterpretedRuntime$InterpretedExecutionPlan.run(InterpretedRuntime.scala:84)
    at org.neo4j.cypher.internal.compatibility.CypherCurrentCompiler$CypherExecutableQuery$$anonfun$execute$3.apply(CypherCurrentCompiler.scala:204)
    at org.neo4j.cypher.internal.compatibility.CypherCurrentCompiler$CypherExecutableQuery$$anonfun$execute$3.apply(CypherCurrentCompiler.scala:190)
    at org.neo4j.cypher.exceptionHandler$runSafely$.apply(exceptionHandler.scala:89)
    at org.neo4j.cypher.internal.compatibility.CypherCurrentCompiler$CypherExecutableQuery.execute(CypherCurrentCompiler.scala:223)
    at org.neo4j.cypher.internal.ExecutionEngine.execute(ExecutionEngine.scala:101)
    at org.neo4j.cypher.internal.javacompat.ExecutionEngine.executeQuery(ExecutionEngine.java:85)
    at org.neo4j.bolt.v1.runtime.TransactionStateMachineV1SPI$BoltResultHandleV1.start(TransactionStateMachineV1SPI.java:189)
    at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State.startExecution(TransactionStateMachine.java:506)
    at org.neo4j.bolt.v1.runtime.TransactionStateMachine$State$2.run(TransactionStateMachine.java:385)
    at org.neo4j.bolt.v1.runtime.TransactionStateMachine.run(TransactionStateMachine.java:110)
    at org.neo4j.bolt.v1.runtime.TransactionStateMachine.run(TransactionStateMachine.java:98)
    at org.neo4j.bolt.v3.runtime.TransactionReadyState.processRunMessage(TransactionReadyState.java:82)
    at org.neo4j.bolt.v3.runtime.TransactionReadyState.processUnsafe(TransactionReadyState.java:49)
    at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48)
    at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.nextState(BoltStateMachineV1.java:144)
    at org.neo4j.bolt.v1.runtime.BoltStateMachineV1.process(BoltStateMachineV1.java:92)
    at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:89)
    at org.neo4j.bolt.runtime.MetricsReportingBoltConnection.lambda$enqueue$0(MetricsReportingBoltConnection.java:68)
    at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:191)
    at org.neo4j.bolt.runtime.MetricsReportingBoltConnection.processNextBatch(MetricsReportingBoltConnection.java:86)
    at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:139)
    at org.neo4j.bolt.runtime.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:171)
    at org.neo4j.bolt.runtime.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$2(ExecutorBoltScheduler.java:154)
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
karrtikiyer commented 4 years ago

Neo Version: 3.5.16 Graph Data Science Library: 1.0.0

DarthMax commented 4 years ago

Hej @karrtikiyer, thank you for reporting the issue. I tried to reproduce the error, but have been unsuccessful so far. I see that you are running version 1.0.0, we have released a couple of new version in the meantime. If you are still running on Neo4j 3.5, could you try to run the algorithm with the 1.1.1 release? If the error persists, would it be possible for you to share your database with us, or another dataset with which you are able to trigger this issue?

Best, Max

omidtavakoli commented 4 years ago

I have the same issue here. I am using Neo4j 3.5.9 and GDS 1.1.1. After running "gds.alpha.ml.ann.stream", I got this error:

Neo.ClientError.Procedure.ProcedureCallFailed: Failed to invoke procedure gds.alpha.ml.ann.stream: Caused by: java.lang.IllegalMonitorStateException

DarthMax commented 4 years ago

@omidtavakoli we are still having a hard time reproducing the error. Could you provide us with a dataset that triggers this error appear? Best, Max

omidtavakoli commented 4 years ago

@DarthMax Of course. This is my procedure for setting up GDS:

1. running neo4j docker and exec into: 

 docker run -d \
--name "neo4j_gds" \
--env NEO4J_AUTH=test/test \
--memory=30G \
--memory-swap=30G \
--cpus=7 \
--env=NEO4J_dbms_memory_heap_initial__size=10G \
--env=NEO4J_dbms_memory_heap_max__size=10G \
--env=NEO4J_dbms_memory_pagecache_size=4G \
--env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
--publish=7478:7474 \
--publish=7691:7687 \
--volume=/import:/import \
--volume=/data:/data \
--volume=/logs:/logs \
neo4j:3.5.9

2. Download GDS 1.1.1 plugin from this url to plugins folder and unzip

https://s3-eu-west-1.amazonaws.com/com.neo4j.graphalgorithms.dist/graph-data-science/neo4j-graph-data-science-1.1.1-standalone.zip

3. update config file with these lines:

dbms.security.procedures.unrestricted=gds.*
dbms.security.procedures.whitelist=gds.*

4. Delete existing data

rm -rf /data/databases/graph.db/*

5. Import data using admin-import. 

bin/neo4j-admin import --ignore-missing-nodes=true \
--nodes /import/movies_header.csv,/import/movies.csv \
--nodes /import/users_header.csv,/import/users.csv \
--relationships /import/users_movies_header.csv,/import/users_movies.csv

users_header.csv:

users_id:ID,:LABEL

users.csv sample

usr1,USER
usr2,USER
usr3,USER
usr4,USER
usr5,USER

movies_header.csv

movie_id:ID,Title,rating_mean:float,:LABEL

movies.csv sample

19522,19522,3.97,MOVIE
22880,22880,4.21,MOVIE
26558,26558,4.25,MOVIE
27238,27238,4.4,MOVIE
17611,17611,4.45,MOVIE

users_movies_header.csv

:START_ID,:END_ID,rating:string,:TYPE

users_movies.csv

usr1,19522,5,WATCHED
usr2,22880,5,WATCHED
usr3,26558,5,WATCHED
usr4,27238,5,WATCHED
usr5,17611,5,WATCHED

6. Change new data ownership

chown -R neo4j:neo4j /data/databases/graph.db/*

7. restart container and checkout localhost:7478

omidtavakoli commented 4 years ago

@DarthMax any update?

omidtavakoli commented 4 years ago

I test without docker on cetos7 server and got the same error: Neo4j 3.5.17 and gds 1.1.1

Failed to invoke procedure gds.alpha.ml.ann.stream: Caused by: java.lang.IllegalMonitorStateException

DarthMax commented 4 years ago

Hej @omidtavakoli! Thank you for providing the instructions. I was finally able to recreate the failure. We were able to identify the bug and I submitted a fix. Once it is reviewed it will appear in this repository and will be included in the next release. If you want to try it before that you can build the project manually. Thanks again! Best, Max

omidtavakoli commented 4 years ago

@DarthMax Thank you for your response. I want to build the project by myself but I cannot find your commit on the master branch and 1.1.1. Please send your commit number or link.

Best, Omid

DarthMax commented 4 years ago

Hej @omidtavakoli, sorry for the delay, it took a while to get the PR reviewed and merged. The fixes are now in with the latests commit being https://github.com/neo4j/graph-data-science/commit/f978700cacf6323000e47a99bdf9c894fbf002b4. Best, Max

omidtavakoli commented 4 years ago

Thank you @DarthMax for following this issue. It's ok now

abydalzain commented 4 years ago

I'm running neo 4.04 and GDS 1.21, and I'm getting the same error.

Mats-SX commented 4 years ago

@abydalzain I think this fix was released in GDS 1.2.2; could you try to upgrade your GDS version and see if that resolves the issue?

abydalzain commented 4 years ago

Thanks. GDS 1.2.2 has the fix