large dataset crashes graph validation

[amnonkhen]

See @idazucchi 's dataset:

• Wrangling ticket ebi-ait/hca-ebi-wrangler-central#853
• ops ticket ebi-ait/hca-ebi-wrangler-central#1050 There are 2 problems:
  1. Hydration from ingest using the uuid fails.
  2. Hydration using the spreadsheet fails while trying to load the records into neo4j. It just never stops, even after a few hours of running.

[amnonkhen]

Graph validation passes if running using on older version of the validator. See Wei's comment

[arschat]

Amnon started test on dev. Method of testing:

1. connect to pod
2. run graph validation command line

   ingest-graph-validator -d localhost hydrate ingest c6ee3bfd-aafa-47d1-bb5d-cb9665b61f7f

   Dataset used: Melanocytes: submission c6ee3bfd-aafa-47d1-bb5d-cb9665b61f7f

Node - dataset is only partially linked. It has 5k links with is nearly 1/20 of the number on the prod dataset.

Test on dev passed successfully.

See ingest-graph-validator#78 for runtime analysis.

Logs:

Click to see logs

``` ❯ kubectl exec -it ingest-graph-validator-7d69d49c7-jjrpz /bin/sh kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead. /ingest-graph-validator # ingest-graph-validator -d localhost hydrate ingest c6ee3bfd-aafa-47d1-bb5d-cb9665b61f7f 23-04-06 21:35:21 [ingest_graph_validator.ingest_graph_validator] - INFO: using neo4j server at localhost:7687 23-04-06 21:35:21 [ingest_graph_validator.ingest_graph_validator] - INFO: Connecting to neo4j... 23-04-06 21:35:21 [ingest_graph_validator.hydrators.hydrator] - INFO: Started ingest hydrator for for submission [c6ee3bfd-aafa-47d1-bb5d-cb9665b61f7f] 23-04-06 21:35:22 [ingest_graph_validator.hydrators.hydrator] - INFO: Found project for submission d501de74-267f-4522-881a-7b0a3f27c20a 23-04-06 21:35:22 [ingest_graph_validator.hydrators.hydrator] - INFO: Found submission for project with uuid c6ee3bfd-aafa-47d1-bb5d-cb9665b61f7f 23-04-06 21:35:22 [ingest_graph_validator.utils] - INFO: [build_entities_from_submission] took [0] ms 23-04-06 21:35:22 [ingest_graph_validator.hydrators.hydrator] - INFO: processing entity type biomaterials 23-04-06 21:35:23 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 0 items of type biomaterials 23-04-06 21:35:25 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 1000 items of type biomaterials 23-04-06 21:35:26 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 2000 items of type biomaterials 23-04-06 21:35:27 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 3000 items of type biomaterials 23-04-06 21:35:28 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 4000 items of type biomaterials 23-04-06 21:35:29 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 5000 items of type biomaterials 23-04-06 21:35:31 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 6000 items of type biomaterials 23-04-06 21:35:32 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 7000 items of type biomaterials 23-04-06 21:35:33 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 8000 items of type biomaterials 23-04-06 21:35:35 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 9000 items of type biomaterials 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 9826 items of type biomaterials 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: processing entity type files 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 0 items of type files 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 96 items of type files 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: processing entity type processes 23-04-06 21:35:36 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 0 items of type processes 23-04-06 21:35:38 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 1000 items of type processes 23-04-06 21:35:40 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 2000 items of type processes 23-04-06 21:35:42 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 3000 items of type processes 23-04-06 21:35:44 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 4000 items of type processes 23-04-06 21:35:46 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 5000 items of type processes 23-04-06 21:35:48 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 6000 items of type processes 23-04-06 21:35:49 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 7000 items of type processes 23-04-06 21:35:51 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 8000 items of type processes 23-04-06 21:35:53 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 9000 items of type processes 23-04-06 21:35:55 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 9852 items of type processes 23-04-06 21:35:55 [ingest_graph_validator.hydrators.hydrator] - INFO: processing entity type projects 23-04-06 21:35:55 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 0 items of type projects 23-04-06 21:35:55 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 1 items of type projects 23-04-06 21:35:55 [ingest_graph_validator.hydrators.hydrator] - INFO: processing entity type protocols 23-04-06 21:36:05 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 0 items of type protocols 23-04-06 21:36:05 [ingest_graph_validator.hydrators.hydrator] - INFO: finished 16 items of type protocols 23-04-06 21:36:05 [ingest_graph_validator.utils] - INFO: [process_submission] took [42953] ms 23-04-06 21:36:05 [ingest_graph_validator.hydrators.hydrator] - INFO: importing nodes 23-04-06 21:36:10 [ingest_graph_validator.hydrators.hydrator] - INFO: imported 19791 nodes 23-04-06 21:36:10 [ingest_graph_validator.utils] - INFO: [get_nodes] took [5479] ms 23-04-06 21:36:10 [ingest_graph_validator.utils] - INFO: [get_edges] took [0] ms 23-04-06 21:36:10 [ingest_graph_validator.hydrators.hydrator] - INFO: filling nodes 23-04-06 21:36:10 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 0 items 23-04-06 21:36:23 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 1000 items 23-04-06 21:36:35 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 2000 items 23-04-06 21:36:48 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 3000 items 23-04-06 21:37:01 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 4000 items 23-04-06 21:37:17 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 5000 items 23-04-06 21:37:30 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 6000 items 23-04-06 21:37:42 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 7000 items 23-04-06 21:37:55 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 8000 items 23-04-06 21:38:07 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 9000 items 23-04-06 21:38:18 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 10000 items 23-04-06 21:38:30 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 11000 items 23-04-06 21:38:43 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 12000 items 23-04-06 21:38:56 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 13000 items 23-04-06 21:39:10 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 14000 items 23-04-06 21:39:24 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 15000 items 23-04-06 21:39:36 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 16000 items 23-04-06 21:39:48 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 17000 items 23-04-06 21:40:02 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 18000 items 23-04-06 21:40:17 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 19000 items 23-04-06 21:40:25 [ingest_graph_validator.hydrators.hydrator] - INFO: filling edges 23-04-06 21:40:25 [ingest_graph_validator.hydrators.hydrator] - INFO: importing edges 23-04-06 21:40:25 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 0 items 23-04-06 21:40:30 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 1000 items 23-04-06 21:40:33 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 2000 items 23-04-06 21:52:31 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 3000 items 23-04-06 21:58:11 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 4000 items 23-04-06 22:03:03 [ingest_graph_validator.hydrators.hydrator] - INFO: get_edges: completed 5000 items 23-04-06 22:22:34 [ingest_graph_validator.hydrators.hydrator] - INFO: importing edges finished 23-04-06 22:22:34 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 0 items 23-04-06 22:22:46 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 1000 items 23-04-06 22:22:59 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 2000 items 23-04-06 22:23:10 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 3000 items 23-04-06 22:23:21 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 4000 items 23-04-06 22:23:33 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 5000 items 23-04-06 22:23:44 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 6000 items 23-04-06 22:23:57 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 7000 items 23-04-06 22:24:09 [ingest_graph_validator.hydrators.hydrator] - INFO: maybe_commit_tx: completed 8000 items 23-04-06 22:24:15 [ingest_graph_validator.hydrators.hydrator] - INFO: hydration finished ```

[amnonkhen]

I deployed to prod and am testing the same dataset (Human Melanocytes 251bf9c5-a5e5-4da6-b2ad-93bac7d47eb4). It is running steadily.

• [ ] watch logs
• [ ] watch monitoring

[amnonkhen]

Further testing on dev using Nephritis dataset It causes "Killed" error in the graph validation pod after 16k biomaterials. See 2 occurrences in the screenshot and logs below. Increasing the pod memory limit solved this problem, although the high memory consumption needs to be dealt with separately. It is due to loading all metadata nodes into a python dict in memory.