Closed lgeistlinger closed 2 years ago
That sounds good. Should we also add code for the wiki to ignore the WikiWorks... users for these fields? Usually our edits are not very scientifically useful so we shouldn't be getting credit :).
Yeah that would make sense to hide the Wikiworks74 etc curations :)
There are some more considerations to this though. From the discussion arising in #93, it becomes clear that we would actually like the exported column "Revision editor" to list the person who marked the content as reviewed as @ftzohra22 points out in https://github.com/waldronlab/BugSigDB/issues/93#issuecomment-880132766.
It seems "Revision editor" is correct, but "Reviewer" could be another column in the export.
Agreed.
Adding a priority label to this as we will make our first versioned data release (https://github.com/waldronlab/BugSigDBExports/issues/4) as soon as it is resolved.
Moving @lwaldron's comment from #93 here:
I also noted that the "Complete/Incomplete" status would be worth exporting too. So the two items here are to include in the export
1. "Reviewed by" (missing or who marked study/experiment/signature as reviewed), and 2. Complete or Incomplete (or TRUE/FALSE)
I'd like to add two more columns in the export:
as it arises from https://github.com/waldronlab/BugSigDB/issues/55#issuecomment-877253482. This will facilitate ontology-based queries downstream of the export in eg bugsigdbr and/or BugSigDBStats.
Sorry. I'm not clear on how to proceed with the WikiWorks user removal. Right now Curator and Revision Editor can list a WikiWorks user. Do we want to change either one or both? And if we are removing the WikiWorks user, is it OK if we end up with these fields as blank?
It seems "Revision editor" is correct, but "Reviewer" could be another column in the export.
Do we also want the "Reviewer" field to be prominently displayed next to the Revision Editor field (on the site itself)? Or is OK that you need to hover over the (!) icon in order to see it?
Sorry. I'm not clear on how to proceed with the WikiWorks user removal. Right now Curator and Revision Editor can list a WikiWorks user. Do we want to change either one or both? And if we are removing the WikiWorks user, is it OK if we end up with these fields as blank?
I don't feel strongly about this, but blank seems equally informative as WikiWorks user, so fine with me. I would want to search for those pages where both curation and revision editor are blank, and at least make sure they are at least reviewed. I don't think it seems worthwhile to dig out again the pre-wiki curators of these pages.
Do we also want the "Reviewer" field to be prominently displayed next to the Revision Editor field (on the site itself)? Or is OK that you need to hover over the (!) icon in order to see it?
Again I don't feel strongly about this. Both seem OK, although displaying the Reviewer may be a bit better to recognize reviewing as an important contribution. FYI in case this helps inform a decision: we currently have many unreviewed pages, and although we will catch up some, reviewing is almost as much work as the original curation and it seems likely that curation will always outpace reviewing.
Complete or Incomplete (or TRUE/FALSE)
This part is now "Complete". The requested column can be found in any CSV.
It seems "Revision editor" is correct, but "Reviewer" could be another column in the export.
Done.
I'd like to add two more columns in the export:
1. The EFO ID of the corresponding condition investigated (such as [EFO:0001075](http://www.ebi.ac.uk/efo/EFO_0001075) for "ovarian carcinoma") 2. The UBERON ID of the corresponding body site sampled (such as [UBERON:0000341](http://purl.obolibrary.org/obo/UBERON_0000341) for "throat")
Please see if this looks OK in the new CSVs.
As noted in #94, it is problematic for downstream applications that these changes to the export files also cause the link locations to change. It would be preferable if we would have stable links such as eg https://bugsigdb.org/export/experiments.csv
that would always point to the latest version of these bulk export files.
Columns State
(for completion status) and Reviewer
(for review status) look great.
EFO / UBERON ID: It looks like https://github.com/waldronlab/BugSigDB/issues/55#issuecomment-877253482 precedes this, as these columns are currently mostly blank. We are missing eg the corresponding EFO ID for terms like "adenoma". This is true for the export files but also the pages itself (eg https://bugsigdb.org/Study_1/Experiment_1), where links to the EFO IDs / EFO pages are missing.
Also, IDs such as EFO:0001075
and UBERON:0000341
are currently abbreviated as 1075
and 341
in the export columns, but I think it would be preferable to have them exported fully spelled out.
I figured out what's happening. The leading zeros are being exported, as you can see in a text editor:
The spreadsheet application is interpreting these as numbers and removing the leading zeros, but it's not the application's fault. Semantic MediaWiki should be surrounding these fields with quotes, since they are stored internally as a text data type. This is likely a bug with Semantic MediaWiki. We'll research this and see what we can do.
My previous comment was incorrect. Semantic MediaWiki is doing everything right. The SMW CSV exporter simply uses the PHP native fputcsv to create its CSV files. And the CSV spec appears to say that numbers don't need to be wrapped in quotes. So the spreadsheet application is what is at fault since technically all CSV fields should be interpreted as text, but I guess it's doing its best to figure out what a user would expect.
If we want to work around this we would need to modify the extension to use our own custom CSV encoder, which would be a bit of a project and probably not a good idea.
Right, the IDs are indeed exported with leading zeros (my bad), but would it be possible to export them with leading "EFO:" for condition and leading "UBERON:" for body site? Not a problem if not, as we can also add them downstream in our R application. It would just be more convenient and more straightforward to use for everyone who doesn't use the R application but rather works on the exported files themselves.
"EFO:" and "UBERON:" are also not a bad idea to include if it's not difficult because there are also other potentially relevant ontologies. We could in theory mix ontologies and use downstream tools to map them, and even if we never do that, it's more communicative to someone downloading the file who isn't familiar with its contents.
It's a great point. As we already observed (https://github.com/waldronlab/BugSigDB/issues/55#issuecomment-801474939), EFO actually already is an umbrella ontology, meaning not all EFO IDs start with "EFO:", but also "CHEBI:", "Orphanet:", "HP:", and "MONDO:", ... In numbers only 11,338 out of 27,175 terms in the EFO start with "EFO:". We thus indeed need the prefixes as available from the Term ID <-> Term name mappings.
We combined the two columns as requested:
Please let me know if this looks OK now.
Looks good! My only question is why so many EFO IDs are missing, since all the condition terms were taken from the EFO?--
Levi Waldron
Associate Professor
Department of Epidemiology and Biostatistics
CUNY Graduate School of Public Health and Health Policy
Institute for Implementation Science in Population Health
55 W 125th St, New York NY 10035
Join the microbiome Virtual International Forum: https://microbiome-vif.org
I also noticed these on CSV exports (my filters were 'streptococcus' 'homo sapiens' and 'feces') :
Two points here:
They come from the glossary. Right now, we'd need to create a new page for each condition (although we could import a CSV). However, this also depends on #89. If we go ahead with that, we'll be able to query their API and remove the need for the Glossary.
I think this is higher priority than #89 and needed for basic functionality, whereas #89 is more of a wishlist. For now, the following would suffice for basic functionality:
Adding the bug label since specifically for the incomplete ontology columns in the export. That is the final remaining item for our first "official" export of bugsigdb, see https://github.com/waldronlab/BugSigDBExports/issues/4#issuecomment-917712225. This is a high priority item.
Hi @tosfos: how do you think we should best proceed for this? Should I provide a csv for importing or do you go ahead with pulling the mappings directly from EFO and UBERON?
Sorry about the delay! We'll send over a CSV shortly with a header row and an example row. That will help you fill it out.
Glossary Import Format.csv Please use the attached CSV. Here are some notes on the fields:
Sounds good. Which field is used to store the term ID (eg "EFO:0001075") for a given term name (eg "ovarian carcinoma")?
We should be able to derive this automatically from the URL. I don't think we need to worry about that field.
Alright. A look at what information is typically available for a term in the official EFO OBO release - here for term "ovarian carcinoma":
[Term]
id: EFO:0001075
name: ovarian carcinoma
def: "A malignant neoplasm originating from the surface ovarian epithelium. It accounts for the greatest number of deaths from malignancies of the female genital tract and is the fifth leading cause of cancer fatalities in women. It is predominantly a disease of older white women of northern European extraction, but it is seen in all ages and ethnic groups. Adenocarcinomas constitute the vast majority of ovarian carcinomas. The pattern of metastatic spread in ovarian carcinoma is similar regardless of the microscopic type. The most common sites of involvement are the contralateral ovary, peritoneal cavity, para-aortic and pelvic lymph nodes, and liver. Lung and pleura are the most common sites of extra-abdominal spread. The primary form of therapy is surgical. The overall prognosis of ovarian carcinoma remains poor, a direct result of its rapid growth rate and the lack of early symptoms. --2002" [NCIT:C4908]
comment: Editor note: unclear why this is distinct from malignant ovarian epithelial tumor in NCIT.
synonym: "carcinoma of ovary" EXACT [MONDO:patterns/carcinoma, NCIT:C4908]
synonym: "carcinoma of the ovary" EXACT [NCIT:C4908]
synonym: "epithelial ovarian cancer" EXACT [NCIT:C4908]
synonym: "ovarian cancer" EXACT [NCIT:C4908]
synonym: "ovarian carcinoma" EXACT [] {comment="Mondo preferred label 15.08.2021."}
synonym: "ovarian carcinoma" EXACT [DOID:4001, NCIT:C4908]
synonym: "ovarian carcinoma" EXACT [] {comment="preferred label from MONDO"}
synonym: "ovarian epithelial cancer" EXACT [NCIT:C4908]
synonym: "ovary carcinoma" EXACT [MONDO:patterns/location]
xref: DOID:4001 {source="EFO:0001075", source="MONDO:equivalentTo"}
xref: EFO:0001075 {source="DOID:4001", source="MONDO:equivalentTo"}
xref: MONDO:0005140
xref: NCIT:C4908 {source="DOID:4001", source="EFO:0001075", source="MONDO:equivalentTo"}
is_a: Orphanet:398934 {source="DOID:4001", source="MONDO:Redundant", source="MONDOLEX:0005140", source="NCIT:C4908"} ! Malignant epithelial tumor of ovary
relationship: EFO:0000784 UBERON:0000992 ! has_disease_location ovary
property_value: closeMatch http://linkedlifedata.com/resource/umls/id/C0677886
property_value: closeMatch http://linkedlifedata.com/resource/umls/id/C0677886
property_value: exactMatch DOID:4001
property_value: exactMatch DOID:4001
property_value: exactMatch NCIT:C4908
property_value: exactMatch NCIT:C4908
property_value: gwas:trait "true" xsd:string
That means I will be able to deliver fields 1-5 of your csv with
(lhs: fields in your csv; rhs: fields in the EFO)
(this also provides for how term IDs should be exported in the ontology columns that we request in the thread above).
We added the Latin column since we assumed it would be a nice feature given the site's medical nature. If it won't be used, we'll remove it. Please advise.
On second thought, you can ignore the Term column. It looks like it will always have the same data as the Page Name, so we can just calculate the lowercase of that field instead.
The latin column sounds indeed like a nice feature, but as EFO/UBERON do not provide latin equivalents for the terms, and I don't see a way of easily obtaining that otherwise, I suggest to just drop that column.
I have accordingly prepared the csvs for:
I expect some hiccups and some manual work when matching those against the terms that are already on bugsigdb.org. The question is whether it's better to clean up offending terms before or after the import (ie those terms that are already on bugsigdb.org but don't have an entry in the import files above).
And just to keep in the back of our heads: EFO and UBERON update on a regular basis, ie we should start thinking at some point about how to automatically keep bugsigdb.org's glossary in sync with EFO/UBERON - but it is my understanding that this will be part of the postponed activities planned in #89.
Thanks. We'll drop the latin column.
Does it make sense to first clean up some of the strange formatting like:
293 cell$;A-293 cell$;A293 cell$;HEK cell$;HEK293 cell$;human embryonal kidney cell$;human embryonic kidney cell
Or is that something we would do after the import?
some manual work when matching those against the terms
If they are valid terms, it makes sense to just add these as aliases, and these can be discovered and fixed after the import. If they are not valid (like a misspelling), they should probably be cleaned up before the import.
it is my understanding that this will be part of the postponed activities planned in #89.
Correct.
As described previously, the synonyms for a specific term can contain all kind of special characters so some improvisation was needed as to what to use for separating synonyms.
This is technically a comma-separated list of synonyms for each term, but as commas and semi-colons are partly contained in the synonyms - I used $;
to separate individual synonyms within the list.
Would this be a separator that you can process or would you need a different separator? (which one)
I could potentially also just replace any existing commas in the synonyms, and provide a comma-separated list of synonyms for each term if that makes processing easier.
If they are valid terms, it makes sense to just add these as aliases, and these can be discovered and fixed after the import. If they are not valid (like a misspelling), they should probably be cleaned up before the import.
@tosfos I have accordingly updated the csvs for:
This required action on the following items on bugsigdb.org for "conditon" - which were not a term title in the efo but either:
(i) represented existing synonyms for an EFO term (eg. "race" for EFO term "ethnic group"), (ii) were added as synonyms to appropriate EFO terms where indicated (eg. "sessile serrated adenoma" for EFO term "Colon Sessile Serrated Adenoma/Polyp"), (iii) were added as independent condition terms not in EFO but another ontology such as CHEBI (eg "medroxyprogesterone acetate"), (iv) or were cleaned up on bugsigdb.org itself (eg. "irritable bowel sydrome" -> "irritable bowel syndrome").
"sessile serrated adenoma"
"oral halitosis"
"sjogren's syndrome"
"substance related disorder"
"titanium dioxide nanoparticles"
"air pollution"
"crohn disease"
"medroxyprogesterone acetate"
"cervical cerclage"
"equol"
"irritable bowel sydrome"
"urinary track infection"
"kidney stone"
"female reproductive organ cancer"
"end stage renal disease"
"hiv/aids pre-exposure prophylaxis"
"race"
"antibiotic"
No such actions were needed for body site / UBERON.
I could potentially also just replace any existing commas in the synonyms, and provide a comma-separated list of synonyms for each term if that makes processing easier.
I discarded this idea, as a quick check of synonyms in EFO revealed a total of 14,344 synonyms containing a comma. This includes numerous instances where replacing a comma would obstruct meaning as eg for chemical notations such as "(3S)-3-(4-hydroxyphenyl)-3,4-dihydro-2H-1-benzopyran-7-ol". I thus suggest going ahead with "$;" as separator in the alias/synonym field if that can be processed on your end.
Both files would thus be ready for import on my end.
Great, @lgeistlinger. From my perspective, #73 getting the dynamic home page onto the home page, followed closely by this issue, are our top priorities.
We tested out the "$;" as a delimiter and it seems to work fine. We did a test import of some of these terms, and you can see 17 new terms as part of the Glossary.
Questions about the aliases.
I noticed these by eye but it would be great to see any unknown Conditions here and be able to tell administrators / reviewers how to avoid entering or at least identify non-standard, ie non-ontology main terms.
Are you referring to terms that are neither "main" terms nor aliases? If so, we already track fields that are set improperly.
Hi @lwaldron any more comments on this from your side? Other than that, I think we can go ahead with the full import @tosfos. We are trying to make the Oct 25 deadline of the Bioconductor release schedule to have ontology columns properly exported and freeze a first release of BugSigDB. Thanks!
Just wanted to quickly check in in here one more time as to whether the Oct 25 deadline is realistic for that? It's no problem at all if we don't make it, then we introduce changes somewhere along the way, but it would make for a nice clean introduction into the Bioc release if we were able to meet that date. Thanks!
I think we're ready to go.
We should be able to make that date. I just need to know how to implement it. I'm not sure I understand your comment in item 2 above. See my question above.
Are you referring to terms that are neither "main" terms nor aliases? If so, we already track fields that are set improperly.
I guess terms that we display and allow curators to enter should only be main terms? Aliases seem more useful for user searches.--
Levi Waldron
Associate Professor
Department of Epidemiology and Biostatistics
CUNY Graduate School of Public Health and Health Policy
Institute for Implementation Science in Population Health
55 W 125th St, New York NY 10035
Join the microbiome Virtual International Forum: https://microbiome-vif.org
@tosfos I'd like to filter signatures exported from BugSigDB by review status for #80. This seems currently not possible as the only column that we have in the export csv files on review status is
Revision editor
.However, the field
Revision editor
has value "WikiWorks743" for reviewed content (see eg https://bugsigdb.org/Study_255) as well as content that still needs to be reviewed (see eg https://bugsigdb.org/Study_400). Can this be changed so that this column is empty or NA in the export for content that still need to be reviewed?