Literature Scrapping - Githubissues

neurocausal / neurocausal_data

The data used by neurocausal

0 stars 1 forks source link

Open vborghe opened 2 years ago

vborghe commented 2 years ago

Ideally, we'd have access to the full text of all published paper of interest.

A few of issues with this:

[ ] how to deal with paywalls?
[ ] how to ensure selection of only clinical papers?
[ ] how to deal with old paper that are available only as scanned document hard to parse

vborghe commented 2 years ago

Check out https://github.com/neuroquery/nqdc = contains a tool to download articles from PubMed Central and extract text, metadata and coordinates.

vborghe commented 2 years ago

Here is the tentative list of terms to be used to define a paper "clinical"

vborghe commented 2 years ago

[x] Threshold at 20 for single clinical word = too many false negatives
[x] Threshold at 10 for single clinical word = too many false positive
[x] Threshold at 16 for single clinical word + 60 sum of clinical words = still a bit too many false positives