Filtering the data to the clinical cohort

[complexbrains]

The process requires filtering the data into a set of papers that are clinically relevant. The data to be filtered consists of two different data sets:

1. Inherited data: The set of papers downloaded and used in NeuroQuery project that contains neuroimaging papers from the diverse research domain.

2. NeuroCausal data: This set of papers was downloaded from PubMed by using specific clinical queries.

The data is structured as:

query-aphasia_neurodegenerative.zip
  └── query-aphasia_neurodegenerative
      ├── articlesets
      │   ├── articleset_00000.xml
      │   └── info.json
      └── articles
          ├── 000
          │   └──pmcid_4382926.xml
          ├── 00a
          │   └──pmcid_8317687.xml
          ├── 00b
          │   └── pmcid_6625472.xml
          ├── ...
          ├── e12
          │   ├── pmcid_8832765.xml
          ├── ...
          │
          └── info.json
          └── subset_allArticles_extractedData
                 ├── authors.csv
                 ├── coordinates.csv
                 ├── info.json
                 ├── metadata.csv
                 └── text.csv

The filtering process will use the clinical keywords based on the discussions here.

[vborghe]

Might be solved by @FrancoisPgm PR, @complexbrains feel free to close here after testing 😉

[vborghe]

@complexbrains here are some suggestions:

• [ ] delete https://github.com/neurocausal/neurocausal_meta/blob/main/filter_clinical.py
• [ ] simplify https://github.com/neurocausal/neurocausal_meta/blob/main/filter_clinical2.py so that
• [ ] current def main(args): becomes something like def co_occ_table and only counts (paper by paper) occurances of all words in that vocabulary + their sum (column called "sum_selected_terms") + total number of words in that paper (called "total_#_words")
• [ ] add one function (that gets call only for the clinical use of the script) that takes the column "total_#_words" and the column "sum_selected_terms" and assigned the label (in the column is_clinical) if percentage of clinical words is => 10% - "total_#_words"*"sum_selected_terms"/ 100 => 10
• [ ] change name from filter_clinical2.py topapers_stats.pyto reflect both usages

[vborghe]

And we probably want to remove this to avoid misunderstandings: https://github.com/neurocausal/neurocausal_data/blob/main/clinical_filter_keywords.csv

[vborghe]

@complexbrains the very last version of the filtering code is the one here uploaded? if not, could you make sure it is?

Possible improvements to restrict to human clinical usable data from Pedro:

• to add exclusion criteria: studies with animal models: “mouse”, “mice”, “rat”, “trout”, “fish”, “Peripheral nervous system”, “rodents” (in the title or threshold for frequency in the text)

For the next filter consider counting the words “brain, cortex, subcortical” in the text and their proportion relative to the other terms. If the proportion is too low, like 1/100 we exclude the paper, because it will not contain any brain information.

[martinesparza]

I have created a draft pull request for this issue which hopefully acts as an initial step to cover exclusion criteria.

[nazdoganci]

Hello,

Just a thought but I'm not sure if this would be feasible. If a particular paper is on humans, it can certainly mention animal studies in their introduction and/or discussion and vice versa. So how about scanning through particularly the methods and the results sections of papers and eliminating papers that have the exclusion criteria terms (i.e. "mouse", "rat", "optogenetics" etc.) in the methods and results sections only?