greenelab / connectivity-search-backend

Django backend for hetnet connectivity search
https://search-api.het.io
BSD 3-Clause "New" or "Revised" License
6 stars 2 forks source link

Switch to using neo4j to load Nodes table #40

Closed dhimmel closed 5 years ago

dhimmel commented 5 years ago

Closes https://github.com/greenelab/hetmech-backend/issues/36

Work in progress: must test locally

dhimmel commented 5 years ago

Database import succeeded locally based on https://github.com/greenelab/hetmech-backend/pull/40/commits/5dcfc8bde69dd9263d05e78d179ab814687b8cb3:

python manage.py populate_database --max-metapath-length=3  --reduced-metapaths --batch-size=12000

_download_hetionet_hetmat(self=<dj_hetmech_app.management.commands.populate_database.Command object at 0x7f3dc0381cf8>) ran in 0:00:00
_populate_metanode_table() ran in 0:00:00
_populate_node_table() ran in 0:00:10
_populate_metapath_table() ran in 0:00:00
_download_path_counts(length=1) ran in 0:00:00
_populate_degree_grouped_permutation_table(length=1) ran in 0:00:00
_download_path_counts(length=2) ran in 0:00:01
_populate_degree_grouped_permutation_table(length=2) ran in 0:00:04
_download_path_counts(length=3) ran in 0:00:07
_populate_degree_grouped_permutation_table(length=3) ran in 0:00:29
_populate_path_count_table() ran in 0:05:22

http://localhost:8000/v1/nodes/ also working for me locally

dhimmel commented 5 years ago

I'll comment here once I've reloaded the production database.

dhimmel commented 5 years ago

Database reloading is now complete:

python manage.py populate_database --max-metapath-length=3  --reduced-metapaths --batch-size=12000
python manage.py database_info

_download_hetionet_hetmat(self=<dj_hetmech_app.management.commands.populate_database.Command object at 0x7ff8757dad30>) ran in 0:00:00
_populate_metanode_table() ran in 0:00:00
_populate_node_table() ran in 0:00:09
_populate_metapath_table() ran in 0:00:00
_download_path_counts(length=1) ran in 0:00:00
_populate_degree_grouped_permutation_table(length=1) ran in 0:00:00
_download_path_counts(length=2) ran in 0:00:00
_populate_degree_grouped_permutation_table(length=2) ran in 0:00:04
_download_path_counts(length=3) ran in 0:00:00
_populate_degree_grouped_permutation_table(length=3) ran in 0:00:38
_populate_path_count_table() ran in 0:20:46