jexp / batch-import

generic csv file neo4j batch importer
https://neo4j.com/docs/operations-manual/current/tools/import/
385 stars 157 forks source link

Import not working #72

Closed Tonkonozhenko closed 10 years ago

Tonkonozhenko commented 10 years ago

Hello! I have problems with import. All I receive is

alex@alex-vm:~/projects/WikiVizualizer$ sh import.sh graph.db page_titles.csv links.csv
Usage: Importer data/dir nodes.csv relationships.csv [node_index node-index-name fulltext|exact nodes_index.csv rel_index rel-index-name fulltext|exact rels_index.csv ....]
Using: Importer batch.properties graph.db page_titles.csv links.csv

Writing Configuration File to batch.properties

Here's my files: nodes.csv

id  title
7   Литва
9   Россия
10  Слоновые
...
~1m nodes

rels.csv

id  id  type
7   4927    links
7   3302    links
7   5077    links
...
~10m rels

batch.properties

use_memory_mapped_buffers=true
neostore.nodestore.db.mapped_memory=100M
neostore.relationshipstore.db.mapped_memory=500M
neostore.propertystore.db.mapped_memory=1G
neostore.propertystore.db.strings.mapped_memory=200M
neostore.propertystore.db.arrays.mapped_memory=0M
neostore.propertystore.db.index.keys.mapped_memory=15M
neostore.propertystore.db.index.mapped_memory=15M
jexp commented 10 years ago

Are they tab or space separated?

Can you try one with only ascii names to see if that's a characterset issue?

Tonkonozhenko commented 10 years ago

They are tab separated. It's impossible to make them in ascii because it's list of pages of russian Wikipedia. I'm running it in virtual machine. Maybe can it be just working slowly?

jexp commented 10 years ago

Works for me with these files:

id  title
7   Литва
9   Россия
10  Слоновые
start   end type
0   1   links
1   2   links
2   0   links

start and end in the relationship-file refer to the row numbers of the nodes-file.

If you want to use your external id, either use id:id (and start:id, end:id) as type or use an index, see readme.

sh import.sh rus.db rus/nodes.csv rus/rels.csv 
Usage: Importer data/dir nodes.csv relationships.csv [node_index node-index-name fulltext|exact nodes_index.csv rel_index rel-index-name fulltext|exact rels_index.csv ....]
Using: Importer batch.properties rus.db rus/nodes.csv rus/rels.csv

Using Existing Configuration File

Importing 3 Nodes took 0 seconds 

Importing 3 Relationships took 0 seconds 

Total import time: 1 seconds 
Tonkonozhenko commented 10 years ago

I tested with some rows as you said. It proceeded. Now I'm testing with bigger amount of data. Thank you very much.