search

VirtualFlyBrain / neo4j2owl

Semantics preserving mapping of OWL 2 EL and Neo4J. Under development, do not use.
Apache License 2.0
6 stars 4 forks source link

KB should be restarted after pipeline is run #25

Closed matentzn closed 4 years ago

matentzn commented 4 years ago

After a while, this shows up in log:

2020-06-28 09:09:41.280+0000 INFO  50 ms: (planning: 43, cpu: 14, waiting: 0) - 1527208 B - bolt-session    bolt    null    neo4j-javascript/1.4.1      client/10.42.72.40:35952    server/10.42.96.200:7687>    - CALL db.indexes() - {} - {}
28/06/2020 12:09:412020-06-28 09:09:41.450+0000 INFO  9 ms: (planning: 0, cpu: 1, waiting: 0) - 652488 B - bolt-session bolt    null    neo4j-javascript/1.4.1      client/10.42.72.40:37376    server/10.42.96.200:7687>    - CALL db.indexes() - {} - {}
28/06/2020 12:09:412020-06-28 09:09:41.634+0000 INFO  1 ms: (planning: 0, cpu: 1, waiting: 0) - 652488 B - bolt-session bolt    null    neo4j-javascript/1.4.1      client/10.42.72.40:37420    server/10.42.96.200:7687>    - CALL db.indexes() - {} - {}
28/06/2020 12:16:432020-06-28 09:16:42.910+0000 INFO  83 ms: (planning: 0, cpu: 2, waiting: 0) - 652488 B - bolt-session    bolt    null    neo4j-javascript/1.4.1      client/10.42.72.40:37466    server/10.42.96.200:7687>    - CALL db.indexes() - {} - {}
28/06/2020 12:23:402020-06-28 09:23:40.772+0000 INFO  16 ms: (planning: 0, cpu: 4, waiting: 0) - 753480 B - bolt-session    bolt    null    neo4j-javascript/1.4.1      client/10.42.72.40:48624    server/10.42.96.200:7687>    - CALL db.indexes() - {} - {}
28/06/2020 13:53:242020-06-28 10:53:24.566+0000 ERROR Worker for session '7cca1fc7-d3ca-4d40-8157-db191024e0f9' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:242020-06-28 10:53:24.692+0000 ERROR Worker for session '07e5c0d1-dc25-45f4-8554-e9fbfc3588d9' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:242020-06-28 10:53:24.776+0000 ERROR Worker for session '5353b5a6-9178-4c29-99be-4a78da58af6b' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
28/06/2020 13:53:24 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
28/06/2020 13:53:24 at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:100)
28/06/2020 13:53:24 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
28/06/2020 13:53:24 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
28/06/2020 13:53:24 at java.lang.Thread.run(Thread.java:748)
28/06/2020 13:53:24 at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
28/06/2020 13:53:242020-06-28 10:53:24.812+0000 ERROR Worker for session '56078db7-1868-4d9c-8583-9ad40642e2c9' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
28/06/2020 13:53:24 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
28/06/2020 13:53:24 at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:100)
28/06/2020 13:53:24 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
28/06/2020 13:53:24 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
28/06/2020 13:53:24 at java.lang.Thread.run(Thread.java:748)
28/06/2020 13:53:24 at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
28/06/2020 13:53:242020-06-28 10:53:24.816+0000 ERROR Worker for session 'e2450b7d-db37-4860-9ad1-acfd60879ad6' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
28/06/2020 13:53:24 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
28/06/2020 13:53:24 at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:100)
28/06/2020 13:53:24 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
28/06/2020 13:53:24 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
28/06/2020 13:53:24 at java.lang.Thread.run(Thread.java:748)
28/06/2020 13:53:24 at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
28/06/2020 13:53:242020-06-28 10:53:24.817+0000 ERROR Worker for session 'e96ef7f2-b8fc-4ce4-84a9-2ae49d03a3f9' crashed. Java heap space
28/06/2020 13:53:24java.lang.OutOfMemoryError: Java heap space
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
28/06/2020 13:53:24 at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
28/06/2020 13:53:24 at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
28/06/2020 13:53:24 at org.neo4j.bolt.v1.runtime.concurrent.RunnableBoltWorker.run(RunnableBoltWorker.java:100)
28/06/2020 13:53:24 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
28/06/2020 13:53:24 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
28/06/2020 13:53:24 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
28/06/2020 13:53:24 at java.lang.Thread.run(Thread.java:748)
28/06/2020 13:53:24 at org.neo4j.helpers.NamedThreadFactory$2.run(NamedThreadFactory.java:109)
matentzn commented 4 years ago

@Robbie1977 and I looked a bit around this issue, but its very difficult to pinpoint. After the pipeline has run, the memory of KB hovers around the maximum possible (37GB); a second trigger of CALL ebi.spot.neo4j2owl.exportOWL() simply crashes the database.. @Robbie1977 has now added some additional GC and OOPS parameters to the container, and we will experiment with that.

matentzn commented 4 years ago

Use the backup version of the kb on a fresh container..

matentzn commented 4 years ago

We need to built a restart into the pipeline.

Robbie1977 commented 4 years ago

Backup version is used after pipeline build