Advanced search with Nexus

[glatard]

Purpose

Provide a way for users to do more complex queries of datasets than the current simple text search.

Context

It would allow the portal to support queries such as:

• List datasets that have derived datasets
• List datasets of type fMRI with at least n subjects
• ...

Possible Implementation

In CONP

• Add an "advanced search" text box to the dataset page
• Text box accepts SPARQL queries
• SPARQL query is sent to Nexus endpoint at https://nexus-research.camh.ca
• Nexus endpoint returns dataset identifiers
• Dataset identifiers are displayed in search result

In Nexus

• Configure CONP workspace for anonymous users
• Automatically synchronize with CONP dataset

Relevant People

@samuel-kerrien @liamocn @3design @xlecours

See also CONP dev meeting on May 6, 2020.

[cmadjar]

I guess this would encompass the following feedback from the webinar form?

"It would be better if more filtering options can be added to sections "Data" & "pipeline" for searching and finding the datasets/tools."

[cmadjar]

@xavier in collaboration with Nexus people to implement that. Can you follow up with them.

Timeline: to be done for the end of August 2020.

[zxenia]

The GUI for SPARQL we can use on the frontend: https://github.com/TriplyDB/Yasgui https://triply.cc/docs/yasgui-api

[cmadjar]

FYI - https://nexus-research.camh.ca/web/admin/public/CONP (CONP results on Nexus)

It would appear that recent datasets cannot be found in that link so automatic updates do not seem to be functional.

[zxenia]

I get 'Could not load project CONP' if I follow the link ...

UPDATE: to see the data one needs to login with ORCID ID

[cmadjar]

Email sent to Adeel to figure out why things are not updated on Nexus. According to the Github webhook, things are being pushed: https://github.com/CONP-PCNO/conp-dataset/settings/hooks/234285152

[cmadjar]

Response from Adeel: The correct endpoint for the Nexus search is https://reservoir.global/admin/Public/CONP, the other one was used for demo and testing only.

[cmadjar]

From Adeel's email concerning the search results issues I noticed when testing their GUI (only one dataset returned when typing keyword Alzheimer instead of 3; no datasets returned when typing keyword MRI...)

I think there are a few options for doing that depending on how you want to search:

• full text search of entire content of each JSON file, then maybe the default elastic search view would be fine. You can access the api as Xavier showed in the slack message. To try from the Nexus UI, go here and try the following query: https://reservoir.global/admin/Public/CONP/nxv:defaultElasticSearchIndex/_search

  
  {
  
  "query": {
  
  "match": {
  
    "_original_source": {
  
      "query": "Alzheimer"
  
    }
  
  }
  
  }

}

- if you want to search specific fields from the Nexus UI, you can also use the Studio we’ve set up (it doesn’t bring back the same results as above because it’s searching specific fields): https://reservoir.global/studios/Public/CONP/studios/https%3A%2F%2Freservoir.global%2Fv1%2Fresources%2FPublic%2FCONP%2F_%2Ff5ace263-241e-4680-9802-4c847e4c0490
![image003](https://user-images.githubusercontent.com/1402456/98153642-82535780-1ea1-11eb-9ef8-275305908dcf.jpg)
- to search specific fields through Elasticsearch api, then we’d need to create a view.

[zxenia]

Notes from the dev meeting on 2020-11-25:

1. Identify several main queries (e.g. a dataset has derived datasets) and investigate how they can be wrapped in a user-friendly interface that doesn't require a user to write sparql query.

2. Ideally in the CONP portal, search results from the Nexus should be linked to the CONP dataset detail page. Currently, they are not linked. Nexus delivers stable and resolvable URIs for each dataset. Each URI resolves to a dataset json metadata in nexus elasticsearch (half DATS half schema.org).

[zxenia]

Notes from the dev meeting on 2020-01-13:

• add a note that the Nexus search is experimental on the SPARQL search page
• the link can be added to the dataset search page

[cmadjar]

@zxenia I believe this is done so I will close the issue.

I will create a new issue for the prepopulated queries we talked about in previous meetings and assign it to you if that's OK with you.