surchs
commented
1 year ago Data Model related issues
Missing conditions. ds000115 has a condit column with what looks like clinical conditions. There are levels in there that we cannot model, but we want to model the other levels. Example:
SCZ-SIB-> probably sibling with schizophrenia -> cannot modelSCZ-> probably participant with schizophrenia -> we can model So what do I do about the things we can't model? Our constraints are:- need to map every unique level in diagnosis column to controlled term, or
- mark the level as "missing value" If those are my only options, I need to label this as "missing". Which is clearly wrong. So we probably want to have another category for "cannot model" or otherwise rethink our annotation approach.
Problem with the way we model assessment tools. Right now we assume that if you have several columns linked to the same assessment tool, if any of these has a missing value, then the participant doesn't "have" the tool. That's not great for several reasons:
- The data owner in their infinite wisdom may have decided to discretize a continuous raw score and spread the information across several columns. Then, by definition, only one column would have a non-missing value
- The fact that the other columns have information still means that there is something to be learned for this participant. The non-missing columns are probably still useful Not saying we should add many complex heuristics to determine presence or absence. Any or All might be enough.
A participant with two conflicting diagnoses is currently hard to model. Example:
ds000144 has SADDx_p for social anxiety at preschool and SADDx_f social anxiety at follow-up, but only one imaging session with an unrelated name. If one is "Yes" and the other "No", how should we display it in our model? This is probably mainly a problem for pure BIDS datasets that are doing these wonky hacks to introduce multi-session information into the phenotypic file even though they are not allowed to. so maybe a good response is to say: can't do it -> use BIDS-pheno or MR_proc.
Controlled Vocabulary related problems
Having to look up the controlled terms by hand is pretty annoying (and probably prone to error). If we turn this into a workflow, it will absolutely have to have the possible values pre-configured.
measurement that isn't a tool. For example: ds000201 has BMI1 for Body Mass Index on session 1. Cool. But cogatlas doesn't have that. Makes sense, it's not a tool. More like "heart rate" or "height". Still, something we'd probably like to annotate. So what should we do with these things? SNOMED has them, but then we mix vocabularies...
missing terms. For assessment tools, there are things that exist in the real world, are assessments, but don't exist in cognitive atlas. Example: ds000201 has HADS_Anxiety which is the "Hospital Anxiety and Depression scale, Anxiety subscale". I can find that in SNOMED: snomed:273524006, even down to the subscale. Hhmm...
clashing abbreviations. Not sure if this is a real problem. But I was looking for the "Beck Depression Inventory" in cognitive-atlas. They don't have it. They do have a "BDI" (the common abbreviation), but it refers to "battelle developmental inventory" - no idea what that is. I guess the main observation here is: we need to pay attention to how we let our users search for terms, because they might have a hard time finding the right term if the vocabulary isn't doing a great job with explicit names / abbreviations.
Continuous values problems
categorical variable encoded with numbers When data owners describe a cateogorical variable with numbers, I cannot:
- easily designate missing values (because their unique value is not listed)
- easily map meaningful numeric values (again, because not listed)
Example: in
ds000144I haveSADDx_pencoding presence of social anxiety. From the data dictionary I can see that it is encoded as - 0: Not Meeting Criteria for SAD at Preschool
- 1: Meeting Criteria for SAD at Preschool
So one fix would be to handle it as a categorical variable on account of the
Levelskey in the data dictionary. If the data dictionary was incomplete or missing, we would have to find another way to address this. This will not be a problem for the annotation tool because the user will first tell us what the column is about, so that we can pick the correct workflow from that information.
need to look inside numeric column to categorize
Some continuous columns you actually want to look inside of. If a column has ages, I don't want to see every unique value of course. But people use continuous values for categories and other other things that aren't very obvious.
For example: in ds000115 there is a numeric column saps7. Most likely that has something to do with the Scale for the Assessment of Positive Symptoms for Schizophrenia. But what does the 7 mean? Would be good to look inside of this column. If we end up turning this process into a bulk tool, there should probably be a way to inspect even numeric columns.
Data quality issues
Tool name not recognizable. Some datasets annotate their data with the measured concept rather than the name of the tool. For example "Handedness" instead of "Edinburgh Handedness Inventory". I'm not sure how to annotate this. Cogatlas does have "concepts" like that (that's the whole purpose of the project), but our data model currently expects the range of an assessment edge to be a specific controlled term of a tool. This is probably more of an issue for "bulk annotation" where the user annotating is not the usually the data owner who would have more insight.
Wrong or conflicting description. Low quality data dictionaries are an issue because now I don't know who to believe. Example:
ds000144 has a column GADDx_p. Lools like generalized anx disorder, yes? But description says: "Separation Anxiety Disorder at Preschool". Most likely the description is incorrect.
Not a huge deal for the annotation tool. Could just have a workflow to change the description. But tricky for the bulk-annotator, because I am not the data owner and I don't know what is correct.
Duplicate columns or leftover stuff. Some participants.tsv files are pretty low quality. For example ds000201 has column SRH5_byScanner.y which is likely just a messy pandas merge leftover duplicate. Nothing to do really, just good to know that we're dealing with this kind of quality (and this is probably not a particularly bad dataset).
Multi-session info in participant.tsv file. This is related to the multiple conflicting diagnosis issue described above. The problem arises from people putting repeated measures in the participants.tsv file by just adding more wide columns. We need to decide what we want to do with this. But it will probably be a lot easier to handle when we are atleast aware of the multiple sessions.
jarmoza
alyssadai
Good datasets are:
Here is the main GDrive spreadsheet with all datasets.
To complete:
a copyedit: a new sheet of the original .tsv in https://github.com/neurobagel/bulk_annotations/issues/2 so that we can use these data for further processingFor reviewer: please take a look at the document and see if you have any notes on how we could make this easier to parse.