search

jexp / batch-import

generic csv file neo4j batch importer
https://neo4j.com/docs/operations-manual/current/tools/import/
385 stars 158 forks source link

Neo4j 2.1.1 cannot upgrade the db #101

Closed eugenem closed 9 years ago

eugenem commented 10 years ago

2.1.1 is crashing when trying to upgrade the generated db (generated by 2.1 branch):

Startup failed: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@29192dd' was successfully initialized, but failed to start. Please see attached cause exception.: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@3a71c265' was successfully initialized, but failed to start. Please see attached cause exception.: 'neostore.nodestore.db' has a store version number that we cannot upgrade from. Expected 'NodeStore v0.A.1' but file is version 'NodeStore v0.A.0'.

stephanf commented 10 years ago

Expected 'NodeStore v0.A.1' but file is version 'NodeStore v0.A.0'   Means upgrading from v0.A.0 to v0.A.1 is not supported. You have to rebuild the database.

Am 03. Juni 2014 um 06:55 schrieb Eugene Shaine Mednikov notifications@github.com:

2.1.1 is crashing when trying to upgrade the generated db (generated by 2.1 branch): Startup failed: Component 'org.neo4j.kernel.impl.transaction.XaDataSourceManager@29192dd' was successfully initialized, but failed to start. Please see attached cause exception.: Component 'org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource@3a71c265' was successfully initialized, but failed to start. Please see attached cause exception.: 'neostore.nodestore.db' has a store version number that we cannot upgrade from. Expected 'NodeStore v0.A.1' but file is version 'NodeStore v0.A.0'. — Reply to this email directly or view it on GitHub.

eugenem commented 10 years ago

The db is generated by batch-import. How can rebuild it?

shree11 commented 9 years ago

Even i'm facing the same issue after generating db using batch-import. So did you find any solution for the above post? If yes, please share it with me.

eugenem commented 9 years ago

I've downgraded to Neo4j 2.0

shree11 commented 9 years ago

You mean you are downgraded from neo4j 2.1.2 to 2.0 ? And which version of batch-import are you using ? What was the problem with neo4j 2.1.2 ?

eugenem commented 9 years ago

Yes, downgraded neo4j. And use latest version of batch-import, but old version was working too.

I don't have any solution for batch-import + neo4j 2.1

shree11 commented 9 years ago

Hi eugenem, you are correct. Even i downgraded the neo4j version from 2.1.2 to 2.0.4. Now I'm able to migrate the data from MySQL to neo4j but i'm facing one issue while importing the nodes and relationships. I was having 10 rows in the node.csv file and 10 rows in the rel.csv file . So when i used the batch-import command to import nodes and relationships into neo4j, it showed "Exception in thread "main" org.neo4j.graphdb.NotFoundException: id=10" . So i manually added one row above the 1st node in my node.csv file and when i run the same batch-import command , i'm able to veiw the nodes and relationships in neo4j browser. What is the logic behind this. I dont wanted to add any row as its not a a part of my MySQL database. Please suggest me what i need to do ? If you want to see my csv files i ll attach them.

shree11 commented 9 years ago

my node.csv is looks like below (tab seperated csv)

node id name 0 100 abc 1 101 adam 2 102 smith 3 103 jhon 4 104 lilly 5 105 susen 6 106 george 7 107 sillo 8 108 taylor 9 109 crook 10 110 ralf

Where "0 100 abc" is manually added row .

jexp commented 9 years ago

The node id's will start from zero unless you provide a id:id header which causes the importer to use your id's

Sent from mobile device

Am 28.07.2014 um 11:11 schrieb shree11 notifications@github.com:

my node.csv is looks like below

node id name 0 100 abc 1 101 adam 2 102 smith 3 103 jhon 4 104 lilly 5 105 susen 6 106 george 7 107 sillo 8 108 taylor 9 109 crook 10 110 ralf

Where "0 100 abc" is manually added row .

— Reply to this email directly or view it on GitHub.

shree11 commented 9 years ago

@jexp , You mean to add header like below.. node id:id name

Is my understanding proper? So while creating the node.csv in mysql itself i need to add the header as id:id ? Please let me know and clarify this doubt.

jexp commented 9 years ago

The header would look like:

id:id<tab>name
0<tab>Foo
1<tab>Bar
shree11 commented 9 years ago

Thanks jexp, now i'm able to import the nodes and relationships into neo4j from Mysql. But i have one small query. In the link " http://maxdemarzi.com/2012/02/28/batch-importer-part-2/ " batch import is explained to import the data from postgresql to neo4j. Also we are creating nodes.csv and relationships.csv where in the relationships.csv has fields like start_node,end_node,type,status. But 1st 3 fields are fine ,why do we need status field? What is that exactly? After importing nodes.csv and relationships.csv into neo4j i'm not able to see the "status" field in the neo4j graph. So what is the importance of that filed? Please clarify this doubt.

jexp commented 9 years ago

Status is just a field, that would end up on the relationship. It is something that in your graph model would qualify the relationship between two nodes.

shree11 commented 9 years ago

But we have a field called "type" which defines the relationship between two nodes. Suppose if we take a "name" field as a node then types becomes the relationship. I'm still under confusion for status field because after importing the nodes and relationship csv files from mysql, i did not see the status filed on neo4j graph.

eugenem commented 9 years ago

Is it possible already to import data into Neo4j 2.1?

jexp commented 9 years ago

Sure, use the 2.1 branch

Am 24.03.2015 um 18:11 schrieb Eugene Shaine Mednikov notifications@github.com:

Is it possible already to import data into Neo4j 2.1?

— Reply to this email directly or view it on GitHub https://github.com/jexp/batch-import/issues/101#issuecomment-85604125.

eugenem commented 9 years ago

Well, I just tried the latest 2.1 branch. It still produces NeoStore v0.A.0 which isn't readable by Neo4j 2.1.

sheymann commented 9 years ago

@jexp Same issue, I can't upgrade the databases from 2.1.x to Neo4j 2.2.0 if they have been generated with the batch importer:

2015-04-08 19:23:00.034+0000 ERROR [o.n.s.d.LifecycleManagingDatabase]: Failed to start database. java.lang.RuntimeException: Error starting org.neo4j.kernel.EmbeddedGraphDatabase, C:\Users\Seb\Documents\Neo4j\2.2\graph.db at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:333) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.EmbeddedGraphDatabase.(EmbeddedGraphDatabase.java:59) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.server.database.LifecycleManagingDatabase$1.newGraphDatabase(LifecycleManagingDatabase.java:44) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:110) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:203) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.desktop.runtime.DatabaseActions.start(DatabaseActions.java:70) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.desktop.ui.StartDatabaseActionListener$1.run(StartDatabaseActionListener.java:61) [neo4j-desktop-2.2.0.jar:2.2.0] at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:251) [na:1.7.0_21] at java.awt.EventQueue.dispatchEventImpl(EventQueue.java:727) [na:1.7.0_21] at java.awt.EventQueue.access$200(EventQueue.java:103) [na:1.7.0_21] at java.awt.EventQueue$3.run(EventQueue.java:688) [na:1.7.0_21] at java.awt.EventQueue$3.run(EventQueue.java:686) [na:1.7.0_21] at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_21] at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76) [na:1.7.0_21] at java.awt.EventQueue.dispatchEvent(EventQueue.java:697) [na:1.7.0_21] at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:242) [na:1.7.0_21] at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:161) [na:1.7.0_21] at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:150) [na:1.7.0_21] at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:146) [na:1.7.0_21] at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:138) [na:1.7.0_21] at java.awt.EventDispatchThread.run(EventDispatchThread.java:91) [na:1.7.0_21] Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.transaction.state.DataSourceManager@3b4225d4' was successfully initialized, but failed to start. Please see attached cause exception. at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:328) ~[neo4j-desktop-2.2.0.jar:2.2.0] ... 22 common frames omitted Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.NeoStoreDataSource@24971670' was successfully initialized, but failed to start. Please see attached cause exception. at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) [neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.impl.transaction.state.DataSourceManager.start(DataSourceManager.java:117) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) [neo4j-desktop-2.2.0.jar:2.2.0] ... 24 common frames omitted Caused by: org.neo4j.kernel.impl.storemigration.StoreUpgrader$UpgradingStoreVersionNotFoundException: 'neostore.nodestore.db' does not contain a store version, please ensure that the original database was shut down in a clean state. at org.neo4j.kernel.impl.storemigration.UpgradableDatabase.checkUpgradeable(UpgradableDatabase.java:86) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.impl.storemigration.StoreMigrator.needsMigration(StoreMigrator.java:158) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.impl.storemigration.StoreUpgrader.getParticipantsEagerToMigrate(StoreUpgrader.java:259) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.impl.storemigration.StoreUpgrader.migrateIfNeeded(StoreUpgrader.java:134) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.NeoStoreDataSource.upgradeStore(NeoStoreDataSource.java:567) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.NeoStoreDataSource.start(NeoStoreDataSource.java:475) ~[neo4j-desktop-2.2.0.jar:2.2.0] at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) [neo4j-desktop-2.2.0.jar:2.2.0] ... 27 common frames omitted

jexp commented 9 years ago

Pushed update to 2.1.8

jexp commented 9 years ago

I also pushed a branch and binary for 2.2

sheymann commented 9 years ago

@jexp thanks for the fast update!