search

hetio / hetionet

Hetionet: an integrative network of disease
https://neo4j.het.io
257 stars 68 forks source link

Providing a dump version of Hetionet #43

Open schatzkara opened 2 years ago

schatzkara commented 2 years ago

Hi! Would it be possible to get a .dump version of Hetionet? I use multiple databases in neo4j and have found that this is easiest for loading new databases. By dump version, I mean one created via the neo4j-admin dump command which has file extension .dump (I can give more details if necessary)

Thanks!

dhimmel commented 2 years ago

Ah the neo4j-admin dump command does look ideal. This command wasn't around I don't think when we initially started providing the neo4j exports.

Note that the production database is still using Neo4j 3.5 (https://github.com/hetio/hetionet/issues/32), so that might also create issues. Will need to check whether neo4j-admin dump exists for 3.5 or look into upgrading again.

I won't have time immediately to look into this, so let us know if you're able to create the dump which we could point other users towards.

schatzkara commented 2 years ago

Thanks for the quick response!

I currently use neo4j 4.2 and have been unable to run Hetionet without the dump file. I am trying to avoid having two versions of neo4j running at once, but could certainly try that.

I am going to keep trying to find work arounds and will certainly let you know if anything works!

dhimmel commented 2 years ago

I am trying to avoid having two versions of neo4j running at once, but could certainly try that.

Ideally we could run the export in the Docker container, so its reproducible and doesn't install an extra version of neo4j on the host system.

yingding commented 2 years ago

I would be interested in a dump version too. I am currently using neo4j 4.1.9 also.

dhimmel commented 2 years ago

I'm able to create the dump file with the following:

docker run \
  --name=hetionet-container \
  --rm \
  --publish=7474:7474 \
  --publish=7687:7687 \
  --volume=$HOME/neo4j/hetionet-data:/data \
  --volume=$HOME/neo4j/hetionet-logs:/logs \
  dhimmel/hetionet:hetionet-v1.0_neo4j-3.5.12 \
  neo4j-admin dump --to=/data/hetionet-v1.0-neo4j.dump

Will look into whether this dump from neo4j 3.5.12 can be loaded into neo4j 4+.

dhimmel commented 2 years ago

Okay dump file located at hetionet-v1.0-neo4j.dump. Download link at:

https://github.com/hetio/hetionet/raw/030e44430f168d934c44860ad5a9512deb7c7175/hetnet/neo4j/hetionet-v1.0-neo4j.dump

I haven't tested neo4j-admin load and whether neo4j 4 can load this file. But hopeful it works.

CC @ldnam. Perhaps you can try importing this file via neo4j desktop and see if it works!

DimitrisAlivas commented 1 year ago

Hi, first of all thank you for your work on HetioNet. I am trying to use the dump you shared, load it in my local Neo4j Desktop version in a new DBMS.

However, I am stuck. It throws some error (a pretty big log file). Is there any update on this?

dhimmel commented 1 year ago

However, I am stuck. It throws some error (a pretty big log file)

Can you provide the error message and and note what version of Neo4j Desktop you're using? Put the log in a code block and if it is too large I'll edit your comment to use <details> so it can be minimized.

DimitrisAlivas commented 1 year ago

Thanks for the blazing fast response.

Neo4j Desktop: Version 1.5.7 (1.5.7.102)

Neo4j Community version: 4.4.16

Expand for Neo4j Desktop Log ``` [2023-01-25 15:47:29.188] [info] Neo4j Download check [2023-01-25 15:47:36.983] [info] Using Java version 11 for Neo4j 4.0.0 [2023-01-25 15:47:40.466] [error] Selecting JVM - Version:11.0.17+8-LTS, Name:OpenJDK 64-Bit Server VM, Vendor:Azul Systems, Inc. Not a valid Neo4j archive: /private/var/folders/xt/z_hpsw257x1b02ydd1wl1zhh0000gn/T/5081ba97f0fdb2535e6b7a29cd8ee30c/dumps/neo4j.dump [2023-01-25 15:47:40.586] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 0) [2023-01-25 15:47:40.586] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 1) [2023-01-25 15:47:40.586] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 0) [2023-01-25 15:47:40.587] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 1) [2023-01-25 15:47:40.587] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 0) [2023-01-25 15:47:40.587] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async neo4jConfFolderPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9497:69) at async neo4jConfPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9507:63) at async Object.getConfig (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:10290:22) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:197:17 at async Promise.all (index 1) [2023-01-25 15:47:40.595] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 0) [2023-01-25 15:47:40.596] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 1) [2023-01-25 15:47:40.598] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 0) [2023-01-25 15:47:40.599] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 1) [2023-01-25 15:47:40.601] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-dc15d3f1-886b-4aff-8fbb-b89b19a7a9cb not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 0) [2023-01-25 15:47:40.602] [warn] Failed to get DBMS path through Relate Error: Relate DBMS with tag neo4j-desktop-dbms-database-2355a562-9f98-47ba-864d-b5060b0c9c38 not found at /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13613:15 at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/monad.js:102:46) at Maybe.getOrElse (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/node_modules/@relate/types/dist/monads/primitive/maybe.monad.js:92:22) at getRelateDbms (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:13612:24) at async neo4jPath (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:9485:22) at async transformLocalDbmsConfiguration (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:54:15) at async transformDbmsToGraph (/Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:147:31) at async Promise.all (index 0) at async /Applications/Neo4j Desktop.app/Contents/Resources/app.asar/dist/main.prod.js:226:21 at async Promise.all (index 1) [2023-01-25 15:48:04.087] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml [2023-01-25 15:48:04.219] [info] Online check response: 200 version: 1.5.7 files [2023-01-25 15:48:44.090] [info] Online check request: https://dist.neo4j.org/neo4j-desktop/win/latest.yml [2023-01-25 15:48:44.262] [info] Online check response: 200 version: 1.5.7 files ```

This is the main.log of Neo4J Desktop when I try to "Create new DBMS from dump"

Screenshot 2023-01-25 at 15 50 49

I also tried creating a fresh DBMS and then importing the dump without success. I also tried with neo4j-admin load without much success.

I'm trying to share the logs specific to this instance but they appear to be empty due to some permission issue.. Trying to resolve this issue and get back to you.

Is there any well-known issue I'm unaware of? Is it the version of Neo4j ?

Any help would be greatly appreciated.

dhimmel commented 1 year ago

I think the "Neo4j DBMS logs" is where the actual error message for loading the dump file is. Is this the file you're referring to as unavailable due to:

the logs specific to this instance ... appear to be empty due to some permission issue.

I'm pretty sure the issue is that the dump was created in neo4j 3.5.12 and the dump files are not compatible across major releases. Ideally, we can create a dump for Neo4j 4, but no timeframe on that.

DimitrisAlivas commented 1 year ago

Yes indeed, that's the file I haven't been able to check yet.

I'm pretty sure the issue is that the dump was created in neo4j 3.5.12 and the dump files are not compatible across major releases.

Yeah, I had similar thoughts/worries. Perhaps, one way out of converting hetionet once again for later versions of Neo4j would be to use the docker image (?)

If I end up doing the conversion for later versions like 4.*.*, I'll open a PR.