Store-utils release 3.5.20 destroys the graph by reconnecting all relationships to node id 0

[arikan]

We get a really weird behavior with store-utils (tried in release 3.5.20 and release 3.5.6). It removes all relationships --millions of them-- and reconnects all to a single node, that is the one with id 0. It basically destroys the entire graph database. This is a Neo4j 3.5.5 database.

@jexp any idea why this might be happening? Thanks!

Running store utils throws error "Maximum id limit for RELATIONSHIP_TYPE_TOKEN has been reached. Generated id 65536 is out of permitted range [0, 65535]." Then, when I set dbms.record_format=high_limit in neo4j.config, store-util runs ok. However, the store util processs reconnects all relationship to a single node (node id 0).

What should be the right approach to deal with this problem?

./copy-store.sh enterprise /Users/arikan/Sites/store-util-3.5.6/graph.db /Users/arikan/Sites/graph-fixed.db  '' '' '' '' true

To use your existing Neo4j 3.3.x installation set NEO4J_HOME to your Neo4j directory. Currently set to: /Users/arikan/Library/Application Support/Neo4j Desktop/Application/neo4jDatabases/database-4a52d26d-7e7c-4c60-bbd5-b90ea4b87722/installation-3.5.5
Usage: copy-store.sh [community|enterprise] source.db target.db [RELS,TO,SKIP] [props,to,skip] [Labels,To,Skip] [Labels,To,Delete,Nodes]

Using: Heap 4G Pagecache 2G Edition 'enterprise' from '/Users/arikan/Sites/store-util-3.5.6/graph.db' to '/Users/arikan/Sites/graph-fixed.db' skipping labels: '', removing nodes with labels: '' rels: '' props '' Keeping Node Ids: true

Please note that you will need this memory (2G + 1G + 4G) as it opens 2 databases one for reading and one for writing.

Copying from /Users/arikan/Sites/store-util-3.5.6/graph.db to /Users/arikan/Sites/graph-fixed.db ingoring rel-types [true] ignoring properties [] ignoring labels [] removing nodes with labels [] keep node ids true
Exception in thread "main" java.lang.RuntimeException: Error starting org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory, /Users/arikan/Sites/store-util-3.5.6
    at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:229)
    at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.newFacade(GraphDatabaseFacadeFactory.java:146)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newDatabase(GraphDatabaseFactory.java:133)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:122)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory$EmbeddedDatabaseCreator.newDatabase(GraphDatabaseFactory.java:178)
    at org.neo4j.graphdb.factory.GraphDatabaseBuilder.newGraphDatabase(GraphDatabaseBuilder.java:210)
    at org.neo4j.graphdb.factory.GraphDatabaseFactory.newEmbeddedDatabase(GraphDatabaseFactory.java:79)
    at org.neo4j.tool.StoreCopy.getHighestNodeId(StoreCopy.java:121)
    at org.neo4j.tool.StoreCopy.copyStore(StoreCopy.java:74)
    at org.neo4j.tool.StoreCopy.main(StoreCopy.java:49)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource@369c9bb' was successfully initialized, but failed to start. Please see the attached cause exception "Maximum id limit for RELATIONSHIP_TYPE_TOKEN has been reached. Generated id 65536 is out of permitted range [0, 65535].".
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:116)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.graphdb.facade.GraphDatabaseFacadeFactory.initFacade(GraphDatabaseFacadeFactory.java:224)
    ... 9 more
Caused by: java.lang.RuntimeException: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@60e21209' was successfully initialized, but failed to start. Please see the attached cause exception "Maximum id limit for RELATIONSHIP_TYPE_TOKEN has been reached. Generated id 65536 is out of permitted range [0, 65535].".
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:450)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    ... 14 more
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine@60e21209' was successfully initialized, but failed to start. Please see the attached cause exception "Maximum id limit for RELATIONSHIP_TYPE_TOKEN has been reached. Generated id 65536 is out of permitted range [0, 65535].".
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:473)
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:111)
    at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:434)
    ... 15 more
Caused by: org.neo4j.kernel.impl.store.id.validation.IdCapacityExceededException: Maximum id limit for RELATIONSHIP_TYPE_TOKEN has been reached. Generated id 65536 is out of permitted range [0, 65535].
    at org.neo4j.kernel.impl.store.id.validation.IdValidator.assertIdWithinCapacity(IdValidator.java:90)
    at org.neo4j.kernel.impl.store.id.IdGeneratorImpl.setHighId(IdGeneratorImpl.java:170)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.setHighId(CommonAbstractStore.java:660)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.rebuildIdGenerator(CommonAbstractStore.java:464)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.makeStoreOk(CommonAbstractStore.java:678)
    at org.neo4j.kernel.impl.store.NeoStores.lambda$makeStoreOk$0(NeoStores.java:459)
    at org.neo4j.kernel.impl.store.CommonAbstractStore.visitStore(CommonAbstractStore.java:973)
    at org.neo4j.kernel.impl.store.NeoStores.visitStore(NeoStores.java:504)
    at org.neo4j.kernel.impl.store.NeoStores.makeStoreOk(NeoStores.java:457)
    at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.start(RecordStorageEngine.java:376)
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:452)
    ... 17 more
    Suppressed: org.neo4j.kernel.lifecycle.LifecycleException: Exception during graceful attempt to stop partially started component. Please use non suppressed exception to see original component failure.
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:464)
        ... 17 more
    Caused by: java.lang.NullPointerException
        at org.neo4j.kernel.impl.storageengine.impl.recordstorage.id.BufferedIdController.stop(BufferedIdController.java:59)
        at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.stop(RecordStorageEngine.java:401)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:460)
        ... 17 more

Here is a view from the store-copy.log

Here is how all roads lead to node id 0:

[Marc-Roth]

Hi @jexp , I've got nearly the same issue, but the log for all relations always looks like:

org.neo4j.unsafe.batchinsert.BatchRelationship@6504a875.create Relationship: 0-[:includes]->0 Node[0,used=false,rel=-1,prop=-1,labels=Inline(0x0:[]),light,secondaryUnitId=-1] not in use

So none of the relations were recreated. I've tested it with the versions 3.5.20, 3.5.19 and even 3.5.6 whereas the result was always the same.

[rlippolis]

If the keep_node_ids property was set to false, this might be caused by the issue fixed with this pull request (I had similar issues): https://github.com/jexp/store-utils/pull/55

[huangzhuxing]

I set keep_node_ids to true, all relational data in the database start and end values ​​are 0

[jexp]

Did you use the most recent release?