search

neo4j-contrib / neo4j-etl

Data import from relational databases to Neo4j.
https://neo4j.com/developer/neo4j-etl/
Other
215 stars 46 forks source link

Can't map postgres to Neo4j graph using Neo4j ETL tool? #88

Open stevesolun opened 2 years ago

stevesolun commented 2 years ago

I am trying to map my Cloudquery database (postgres) to Neo4j using ETL Tool.

Project created with the following version:

Version 4.4.5 Neoj4 Desktop version 1.4.15. I am on OSX Monterey 12.5 (21G72), Apple M1 Pro.

For the postgres (where cloudquery is pouring the data) I am using postgres:13.7-alpine3.16 docker image.

Selected RDBMS instance (connection succeeded) and my project. Within the main.log I get:

[2022-07-28 14:27:13.192] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:27:13.844] [info] Process [87889] exit with code '1', signal 'null'
[2022-07-28 14:27:43.264] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:27:43.372] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:28:23.273] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:28:23.382] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:03.278] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:03.386] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:39.909] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.rdbms.Support, jdbc:postgresql://localhost:5432/postgres?ssl=false, postgres, pass'
[2022-07-28 14:29:40.771] [info] Process [88036] exit with code '0', signal 'null'
[2022-07-28 14:29:43.282] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:29:43.338] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:29:44.217] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.NeoIntegrationCli, generate-metadata-mapping, --rdbms:url, jdbc:postgresql://localhost:5432/postgres?ssl=false, --rdbms:password, pass, --rdbms:user, postgres, --output-mapping-file, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.214] [info] Process [88039] exit with code '0', signal 'null'
[2022-07-28 14:30:04.216] [info] Executing '/Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/distributions/java/zulu11.54.25-ca-jdk11.0.14.1/bin/java, -cp, /Users/stevesolun/Library/Application Support/Neo4j Desktop/Application/graphApps/_global/neo4j-etl-ui/dist/neo4j-etl.jar, org.neo4j.etl.util.FileUtils, readfile, /var/folders/nl/301vjtr53s92b4wr66tchrjr0000gn/T/postgresql_postgres_mapping.json'
[2022-07-28 14:30:04.616] [info] Process [88058] exit with code '143', signal 'null'
[2022-07-28 14:30:23.288] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:30:23.596] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:03.292] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:03.344] [info] Online check response: 200 version: 1.4.15
file
[2022-07-28 14:31:43.296] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml
[2022-07-28 14:31:43.459] [info] Online check response: 200 version: 1.4.15

It seems I do get the mapping file but the mapping process takes ages without finishing.

Please advise what should I do to make it work?

Can I upload manually the mapping file? How can I force it to do the mapping? Is it ok it takes so long?

stevesolun commented 2 years ago

@nossila @akollegger @jexp @sarmbruster can you please help me out with it?