52 parsing errors related to mentions

[martinpopel]

While working on a paper about mismatches between coreference and dependency annotations, I've annotated 100 sentences from GUM 2.7 with such mismatches, i.e. with mention spans not forming a catena (treelet) in the UD dependency tree. Here GUM-noncatena-mentions.txt is the relevant part of my annotation:

• 13 sentences were OK because the mismatches were caused by expected cases of coordination and flat structures, so I exclude these 13 sentences from the attached file.
• 52 sentences are annotated as WRONGTREE, i.e. I think there is an error in the dependency tree.
• 22 sentences are annotated as WRONGSPAN, i.e. I think there is an error in the mention annotation (Entity= in the MISC column).
• 15 sentences are annotated as OK-AMBIG, i.e. it can be WRONGTREE or WRONGSPAN.

Of course, my annotation may be wrong, I am not a native speaker. In some sentences, I included an explanation using # Comment =.

My original plan was to further analyze the errors and start a new issue for each type of error and update the annotations to the whole GUM (not only the sample of 100 sentences) and filter out automatically the OK-COORD and OK-FLAT cases... but it seems I won't have time for this soon (and it is the reason I am postponing this issue for several month). That said, I can provide some help (at least the scripts for detecting the mismatches) if anyone is interested.

[amir-zeldes]

Thanks @martinpopel - I saw the paper and was somewhat encouraged that GUM had the lowest conflict score among the corpora, with only 1.5% of mentions being non-catenas :) but given that 75% of those seem to contain some error (syntax or entity span), there is room for improvement, so I am interested in correcting these of course.

That said, we are now very close to releasing a first version of GUM8, which has both a large amount of corrections in both layers (UD+entities/coref), and new data (close to +30K tokens) for which we do not have your output, so some of your issues will have already been caught, while other new ones will be added as well.

If you can provide this kind of output automatically then I am happy to look into it in the coming semester - maybe you can give me updated statistics once GUM 8.0 is in dev?

Thanks!