Open kermitt2 opened 6 years ago
Right. Our instructions are to copy the software name from the text, but looks like that isn't always happening. We should re-emphasize this and add this to our tests. For existing work we could use this file to have students work through and do fixes.
With the news csv files (and ignoring duplicate identifiers), here is an updated version of the list of mention/context mismatches for the five annotation types (software name, version date, version number, creator and url).
unmatched-creator-mention-context.txt unmatched-software-mention-context.txt unmatched-url-mention-context.txt unmatched-version-date-mention-context.txt unmatched-version-number-mention-context.txt
I obtain these ratios:
Unmatched software mentions: 51 out of 5047 total software mentions (1.01%)
Unmatched version number mentions: 26 out of 1682 total version number mentions (1.55%)
Unmatched version date mentions: 14 out of 162 total version date mentions (8.64%)
Unmatched creator mentions: 50 out of 1709 total creator mentions (2.93%)
Unmatched url mentions: 17 out of 282 total url mentions (6.03%)
Total unmatched mentions: 158 out of 5121 total mentions (3.09%)
I think fixing these issues first would be nice because:
from the machine learning learning perspective, we will learn as many wrongly not annotated examples, which will have a big impact on a supervised model,
it impacts artificially inter-agreement evaluation, which is something important to use to estimate the quality of the data set and to spot problems.
Hi @kermitt2 @jameshowison I want to update a bit on my current progress of checking mismatches. So far I've gone through the 432 mismatches between software_name and context strings. I manually categorized each one and in general there're several classes of errors listed below. I've also marked the approximate count of each class of error here so you can get some sense of the magnitude of each issue. But they are estimates as I cannot guarantee I did not make any error when manually categorizing the 432 cases.
So generally these mismatches are instances of:
With James's advice, I added tei_full_quote to all the 432 cases in .ttl, no matter it's the full_quote or tei_full_quote that looks weird. For the cases of which the tei_full_quote that looks weird, I haven't modified the software_name label in .ttl yet. As I am not sure which one to follow. Is there any automated ways to fix them? And may we have systematic ways to avoid these listed issues?
Definitely can give clearer clarifications or examples if these descriptions are dry
I am gonna check the mismatches of other fields in software mentions in the next few days.
It's not always easy to match
software_name
andquote
(as present in the .csv files).After applying some basic soft matching, I still have 138 distinct software names not matching the provided quote nor the actual PDF content (out of 5684 software names present in the .csv file if I am not wrong). Usually this is due to some typo and wrong spelling, for instance:
provided software name: CLUSTER W2 actual context: "and aligned using CLUSTAL W2 software (www.ebi.ac.uk/Tools/msa/clustalw2/) (Fig. 2)."
provided software name: Integrative Genome Viewer actual context: "All candidate variants were visually inspected in Integrative Genomics Viewer.""
Sometimes, it is a bit more complicated, for instance the software name here:
It is easy to fix problems related to casing and missing space, but mismatches as above are complicated to recover.
Here is a file with the 138 problematic software names and their context mention. Note that there are many missing quotes in the .csv file and some missing software names which have not been taken into account in this list (I only used the .csv files so far).
matching_issues_mention_context.txt