Update the MNI-PD graph-ready data and make it federation-accessible

[alyssadai]

(Aka, second iteration of https://github.com/neurobagel/project/issues/145)

To populate a new MNI node(s), we need to generate new versions of the JSON-LD files for the PPMI and QPN that reflect the latest changes to 1) the phenotypic TSVs 2) the Neurobagel data model.

Steps to implement

• [x] Obtain updated phenotypic bagels for PPMI and QPN
  • [x] PPMI (under /data/pd/ppmi-new/tabular/)
  • [x] QPN
• [x] Check that the existing data dictionaries are still in alignment with the updated TSVs, make changes if not
  • [x] PPMI
  • [x] update TermURLs for age heuristic and healthy controls
  • [x] QPN
  • [x] update TermURLs for age heuristic and healthy controls
• [x] Rerun CLI on new TSVs and BIDS datasets
  • [x] PPMI
  • [x] QPN
• [x] Ensure a copy of output JSONLD files are stored in a central directory somewhere on the BIC
  • /data/origami/neurobagel_graph_data
• [x] Upload data to mni_ppmi and mni_qpn databases on st-viateur machine GraphDB server:
  • [x] PPMI
  • [x] QPN
  • NOTE: To do this from the BIC, added a new OpenStack security group to instance to allow auth'd external access to GraphDB server
• [x] Ensure that Neurobagel vocab .ttl is added to both graphs
• [x] Relaunch the federation API with the federation nodes set to the API URls for the open_neuro, mni_qpn, and mni_ppmi databases

[alyssadai]

For reviewer: you can check that the above steps are complete by (in no particular order, the last one being probably the most exciting)

• [ ] Checking the JSONLD files under /data/origami/neurobagel_graph_data and doing e.g., a within-file search for the updated prefix we're using for healthy controls, ncit
• [ ] Checking the value of NB_NODES in the .env file under ~/projects/federation-api on the fairmount machine and the names/ports of the running docker containers (for the PD data) on the st-viateur machine
• [x] Checking that the # triples in the QPN and PPMI graph databases is >0:

  curl -u "USER:PASSWORD" http://206.12.89.194:7200/repositories/mni_ppmi/size
  curl -u "USER:PASSWORD" http://206.12.89.194:7200/repositories/mni_qpn/size

• [x] Running a sample query using the interactive API docs for each PD graph at http://206.12.89.194:8888/docs (PPMI) and http://206.12.89.194:8080/docs (QPN)
  • [x] to check that the Neurobagel vocab file was added successfully, you can execute a request to the /attributes/ endpoint in each API
  • [x] Running a query for "Parkinson's disease" using our deployed query tool and see that (1) the results include both PPMI and QPN 🎉 (2) the participant-level TSVs for both show protected for all expected columns (aside from session path, to be fixed in https://github.com/neurobagel/api/issues/210

[surchs]

@alyssadai thanks a lot for all this work!

I have some questions that should be quick to clarify before we can close

JSONLD files under /data/origami/neurobagel_graph_data check for healthy controls: ncit

I can find ncit prefixes for PPMI, but not for QPN. That seems unintuitive to me. Could you confirm that grep ncit /data/origami/neurobagel_graph_data/qpn/qpn.jsonld should only hit the @context in QPN because

• [ ] no subject in QPN has been labeled with healthy control

in the .env file under ~/projects/federation-api on the st-viateur machine

OK, this made me scratch my head for a bit. Let me try to describe what I think is happening here and you tell me if that's right:

• [ ] the federation API is running on fairmount (206.12.99.17) on port 8888, not on viateur (206.12.89.194)
• [ ] the .env file you describe thus lives on fairmount too and points to nodes on viateur
• [ ] the QPN and PPMI APIs are running on viateur on ports 8888 and 8080 respectively

I think this shows that keeping names of servers, docker containers and numbers of ports clean and organized can quickly get tricky. Let's have a chat about how we can clean up the servers a bit and maybe make deployments more automatic.

Running a sample query using the interactive API docs for each PD graph at http://206.12.89.194:8888/docs (PPMI) and http://206.12.89.194:8080/docs (QPN)

This works well :tada:

query for "Parkinson's disease" and see both PPMI and QPN 🎉

:tada:

So take another look at these healthy_control labels and see if the rest makes sense. Then I think we can close!

[alyssadai]

I can find ncit prefixes for PPMI, but not for QPN. That seems unintuitive to me. Could you confirm that grep ncit /data/origami/neurobagel_graph_data/qpn/qpn.jsonld should only hit the @context in QPN because no subject in QPN has been labeled with healthy control

This is expected, and is due to the issue of the CLI currently only storing info about the first instance of a column about a given Neurobagel variable. The QPN TSV contains two columns about diagnosis, one of which has values PD and PSP (progressive supranuclear palsy), and the other column classes individuals as PD or healthy control. Both were annotated in the data dictionary, but only the PD/PSP diagnosis column is in the graph right now due to order of appearance in the raw data.

• the federation API is running on fairmount (206.12.99.17) on port 8888, not on viateur (206.12.89.194)
• the .env file you describe thus lives on fairmount too and points to nodes on viateur
• the QPN and PPMI APIs are running on viateur on ports 8888 and 8080 respectively

Ah yes, my point about the federation API running on st-viateur was a typo, I meant to say that the new PD data graphs live on that machine. 🙂 Below is the accurate info (most of what you said was correct):

• the federation API is running on fairmount (206.12.99.17) on port ~8888~ 8080, not on viateur (206.12.89.194)
• the .env file you describe thus lives on fairmount too and points to the QPN and PPMI nodes on viateur, and the OpenNeuro node on fairmount
• the QPN and PPMI APIs are running on viateur on ports 8888 and 8080 respectively

[surchs]

Ah, cool. Thanks for clarifying. I copied your explanation into https://github.com/neurobagel/bagel-cli/issues/224 for additional reference.

I think with this we can close this :cook: