search

neo4j-devtools / neo4j-desktop

The Neo4j Graph Platform, on your desktop OS.
19 stars 1 forks source link

neo4j Desktop 4.1.x database failure (on Windows) #90

Closed folterj closed 3 years ago

folterj commented 4 years ago

Using neo4j Desktop on Windows 10 64-bit, unable to upgrade from version 4.0.4 to version 4.1.0 or 4.1.1.

This applies to upgrading an exiting db from 4.0.x to 4.1.x, or loading a dump file or importing from cypher directly into blank 4.1.x. The same content runs fine in neo4j 4.1.1 enterprise server. The only condition we found that caused the issue, was a number of nodes of 2m / 4m labels [edited]. Also happy to provide log files etc. The neo4j log files don't appear to show relevant information, but the large Windows log file shows (part of a) stack trace, beyond the below extract.

Java exception error:

2020-07-08 12:56:27.214+0000 INFO Starting...
2020-07-08 12:56:29.822+0000 INFO ======== Neo4j 4.1.0 ========
2020-07-08 12:56:34.565+0000 INFO Sending metrics to CSV file at C:\Users\Y.Neo4jDesktop\neo4jDatabases\database-9f7e3ce9-0d72-4c2e-a59c-2002ef1a1073\installation-4.1.0\metrics
2020-07-08 12:56:34.607+0000 INFO Bolt enabled on localhost:7687.

A fatal error has been detected by the Java Runtime Environment:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff8199a4323, pid=16752, tid=6852

JRE version: OpenJDK Runtime Environment (Zulu11.35+13-CA) (11.0.5+10) (build 11.0.5+10-LTS)
Java VM: OpenJDK 64-Bit Server VM (11.0.5+10-LTS, mixed mode, tiered, compressed oops, g1 gc, windows-amd64)
Problematic frame:
V [jvm.dll+0x6a4323]

No core dump will be written. Minidumps are not enabled by default on client versions of Windows

An error report file with more information is saved as:
C:\Users\Y\AppData\Local\Temp\hs_err_pid16752.log

If you would like to submit a bug report, please visit:
http://www.azulsystems.com/support/

Small extract from mentioned Java log file:

...
Internal exceptions (20 events):
Event: 11.007 Thread 0x0000020a6ea90800 Exception <a 'java/lang/NoSuchMethodError'{0x00000000de8cde08}: 'java.lang.Object java.lang.invoke.DirectMethodHandle$Holder.invokeInterface(java.lang.Object, java.lang.Object, java.lang.Object, java.lang.Object)'> (0x00000000de8cde08) thrown at [c:/jenki
Event: 11.044 Thread 0x0000020a0faf0000 Exception <a 'java/lang/NullPointerException'{0x00000000de16d6b0}> (0x00000000de16d6b0) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/interpreter/linkResolver.cpp, line 1358]
Event: 11.044 Thread 0x0000020a0faf0000 Exception <a 'java/lang/reflect/InvocationTargetException'{0x00000000de16dc28}> (0x00000000de16dc28) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/runtime/reflection.cpp, line 1246]
Event: 11.045 Thread 0x0000020a0faf0000 Exception <a 'java/lang/reflect/InvocationTargetException'{0x00000000de1738f8}> (0x00000000de1738f8) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/runtime/reflection.cpp, line 1246]
Event: 11.144 Thread 0x0000020a0faef800 Exception <a 'sun/nio/fs/WindowsException'{0x00000000dd2e0218}> (0x00000000dd2e0218) thrown at [c:/jenkins/workspace/zulu11-build-win64/zulu-src/src/hotspot/share/prims/jni.cpp, line 615]
...

Related information: https://community.neo4j.com/t/upgrade-fails-from-enterprise-4-0-4-to-4-1-0/20899/2

nglgzz commented 3 years ago

Hey @folterj, is the upgrade working properly with a smaller graph? From the description and a quick search about the error, it sounds like it could be a memory issue.

Could you check how much memory is Neo4j using right before that error is thrown? Also, does it help increasing the dbms.memory.heap.max_size value in your neo4j.conf file?

folterj commented 3 years ago

Thank you Angelo @nglgzz!

It appears this is now solved.

Things we tested:

Memory:

Using neo4j, Docker, and our own Java core code, optimising memory allocation gets quite complex, so we're relying on neo4j's auto configuration, which has worked better than trying to manually configure it. On a desktop with 16GB RAM, the idle spun up usage is about 2GB.

However it seems that neo4j Desktop (in contrast to neo4j server) automatically sets low limits by default: dbms.memory.heap.max_size=1G dbms.memory.heap.initial_size=512m dbms.memory.pagecache.size=512m

Whereas allowing neo4j to auto-configure - which is what we want and were expecting - sets: dbms.memory.heap.max_size 4278190080 bytes (4.3GB) dbms.memory.heap.initial_size 268435456 bytes (0.2GB) dbms.memory.pagecache.size 6416367616 bytes (6.4GB)

Some unresolved mysteries:

Thank you again for your help, I will post this on the neo4j community site as well.

nglgzz commented 3 years ago

Happy to hear it's solved!

For your first two questions, it's best to ask in the neo4j/neo4j repository. I'm not well versed on the DBMS internals, but my guesses would be:

For the last question, those are the default values to reduce the memory footprint of Neo4j Desktop. These should cover most development cases, but of course, you're welcome to change them if they don't work for you.